Files
nofx/store/store.go
T
tinkle-community 9c5c976d9a feat: Claw402 x402 payment provider + Telegram agent + x402 refactoring (#1409)
* feat(telegram): add AI agent bot with streaming and account context

- Add Telegram bot with long-polling and AI agent loop (api_call tool)
- SSE streaming with real-time message editing and  placeholder
- Account state injection at conversation start (models, exchanges,
  strategies, traders, per-trader PnL and statistics)
- Lane semaphore per chat serializes concurrent messages (60s timeout)
- Idle timeout watchdog (60s) prevents hung streaming connections
- Look-ahead buffer prevents partial <api_call> tag leaking to user
- Fix PUT /strategies/:id to merge config (read-then-merge pattern)
- Add route registry with full API schema for LLM documentation
- Add TelegramConfig store and Web UI config modal
- Add GetAnyEnabled to AIModel store for bot LLM client selection

* fix(telegram): eliminate narration, add full-setup workflow and tests

- Rewrite NO NARRATION rule: response is EITHER api_call tag alone OR
  final text reply — no text before api_call under any circumstances
- Ban all narration patterns: 现在我将/好的/正在/I will/Let me etc.
- Add 'create strategy + create trader + start' full setup workflow
- Add 12 automated tests covering:
  - No narration leaking to user (5 narration variants tested)
  - api_call tag never leaks to user
  - Full setup workflow: POST strategy → verify → POST trader → start
  - Start existing trader workflow
  - Max iterations safety, tag stripping, parser edge cases

* refactor(agent): replace XML api_call with native function calling

Migrate the Telegram bot agent from an XML tag hack (<api_call>) to
OpenAI-native function calling via CallWithRequestFull.

Key changes:
- mcp/interface.go: add parseMCPResponseFull to clientHooks interface
- mcp/client.go: route callWithRequestFull through hooks for overridability
- mcp/claude_client.go: override parseMCPResponseFull for Claude response
  format (tool_use blocks instead of choices[].message.tool_calls)
- telegram/agent/agent.go: rewrite Run() to use CallWithRequestFull;
  define api_request tool with JSON Schema; implement tool-call loop
  with role="tool" result messages; remove XML parsing entirely
- telegram/agent/apicall.go: remove parseAPICall (dead code)
- telegram/agent/prompt.go: simplify — remove XML format instructions,
  replace with concise api_request tool usage instructions
- telegram/agent/agent_test.go: rebuild all tests using LLMResponse
  objects; add TestNarrationStructurallyImpossible, TestOnChunkCalledWithFinalReply,
  TestToolCallIDPropagated; remove XML-specific tests

Architecture advantage: with native function calling, the LLM returns
EITHER ToolCalls OR Content — never both. Narration is now structurally
impossible at the protocol level, not just enforced by prompt rules.

All 11 agent tests pass. mcp package tests pass.

* refactor(mcp): route buildRequestBodyFromRequest through hooks + full Anthropic format

Problem: callWithRequest/Full/Stream all called client.buildRequestBodyFromRequest
directly (not via hooks), so ClaudeClient could never override it. This meant
tool calling sent OpenAI format to Anthropic (wrong field names, wrong roles).

Changes:

mcp/interface.go
- Add buildRequestBodyFromRequest(*Request) map[string]any to clientHooks
- Improve comments: document what each hook group does and why

mcp/client.go
- All three paths (callWithRequest, callWithRequestFull, CallWithRequestStream)
  now call client.hooks.buildRequestBodyFromRequest — ClaudeClient picks up

mcp/claude_client.go
- Full rewrite with format comparison table in package doc
- buildRequestBodyFromRequest: produces correct Anthropic wire format
    * system prompt → top-level "system" field
    * tools: parameters → input_schema, no "type:function" wrapper
    * tool_choice "auto" → {"type":"auto"} object
    * assistant tool calls → content[{type:tool_use, id, name, input}]
    * role=tool results → role=user content[{type:tool_result,...}]
    * consecutive tool results merged into single user turn
- convertMessagesToAnthropic: handles all three message types
- parseMCPResponseFull: extracts text + tool_use blocks
- parseMCPResponse: delegates to parseMCPResponseFull

All mcp and agent tests pass.

* fix(telegram): fix claude client dispatch + strategy creation workflow

- telegram/bot.go: clientForProvider now returns NewClaudeClient() for
  'claude' provider (was incorrectly falling back to DeepSeekClient which
  uses OpenAI wire format, breaking Anthropic API calls)

- api/server.go: fix scan_interval_minutes schema default (3, not 60);
  POST /api/strategies now clearly states config is OPTIONAL with complete
  working defaults; POST /api/traders removes redundant GET workflow note

- telegram/agent/prompt.go: simplify strategy creation — just POST {name}
  without config (backend applies full working defaults automatically);
  only include config when user requests custom settings

* test(mcp): add ClaudeClient wire format tests

Tests cover all Anthropic-specific format conversions:
- system prompt lifted to top-level field
- tools use input_schema (not parameters)
- tool_choice is object {type:auto} not string
- assistant tool calls → content[{type:tool_use}]
- consecutive tool results merged into single user turn
- parseMCPResponseFull: text, tool_use, and error cases
- x-api-key header (not Authorization: Bearer)
- /messages endpoint URL

* fix(telegram): clientForProvider returns correct client for all 7 providers

Previously qwen/kimi/grok/gemini all fell back to DeepSeekClient.
Each provider now gets its own dedicated client with correct default
base URL and model. All 7 providers now fully supported:
openai, deepseek, claude, qwen, kimi, grok, gemini

* fix(telegram): newLLMClient uses bound user's model, not any user's model

GetAnyEnabled() searched across all users in DB — if user B has an
enabled model, bot could use their API key while acting as user A.

Now uses GetDefault(botUserID) which only looks up the bound user's
enabled model, matching the same user scope as all API calls.

* fix(auth): single-user deployment by default, no open registration

Registration logic redesigned:
- Empty DB (first-time setup): registration always open, no config needed
- After first user exists: registration closed by default
- Multi-user opt-in: set REGISTRATION_ENABLED=true + MAX_USERS=N in .env

Config defaults changed:
- RegistrationEnabled: true → false (closed after first user)
- MaxUsers: 10 → 1 (single-user deployment default)

This eliminates the confusion of multiple users appearing in a personal
deployment where Telegram is bound to a single admin account.

* feat(solo): beginner-friendly onboarding — smart setup guide + direct config commands

start.sh:
- Interactive Telegram Bot Token prompt on first run
- Token format validation (must match 12345:ABC... pattern)
- Friendly step-by-step startup instructions after launch

telegram/bot.go:
- /start now shows context-aware setup guide based on actual config state:
  - No AI model → explains how to configure, lists all providers
  - AI model OK but no exchange → guides to configure exchange via chat
  - All configured → full capabilities welcome message
- New: direct setup commands ('配置 deepseek sk-xxx') bypass LLM entirely
  so AI model can be configured even before any model exists (bootstrap fix)
- All messages now in Chinese (匹配用户语言)

telegram/agent/prompt.go:
- Added first-time setup detection section
- Agent told to never ask user to visit web UI — everything via chat

* feat(i18n): bilingual EN/ZH setup guide with language selection

store/telegram_config.go:
- Add Language field to TelegramConfig (persisted in DB)
- Add SetLanguage(lang) and GetLanguage() methods
- Default language: English (en)

telegram/bot.go:
- First /start triggers language selection (1=English, 2=中文)
- /lang command to change language at any time
- awaitingLang state machine handles language choice before any other input
- buildSetupGuide() now fully bilingual (EN/ZH), context-aware:
  Step 1: configure AI model (no model yet)
  Step 2: configure exchange (model OK, no exchange)
  Ready: show full capabilities
- tryHandleSetupCommand() bilingual: 'configure/配置 <provider> <key>'
- helpMessage(lang) fully bilingual
- All error/status messages bilingual

Default: English. isLangDefault() detects whether user has explicitly
chosen a language vs falling back to the 'en' default.

* fix(telegram): use Markdown rendering + simplify language selection condition

- sendMarkdownMsg() helper: sends with ParseMode=Markdown, falls back to plain text
- All formatted messages (langSelectionMsg, buildSetupGuide, helpMessage) now render
  bold text and code blocks correctly in Telegram
- Simplify /start language check: isLangDefault(st) alone is sufficient
  (lang == 'en' && isLangDefault was redundant — GetLanguage returns 'en' when empty)

* fix(start.sh): translate all user-facing text to English

Entire script was in Chinese. Now English-first throughout:
- startup banner, prompts, success/error messages
- setup_telegram(): English instructions and validation messages
- start(): English next-steps after launch
- stop/restart/clean/update/regenerate-keys/show_help: all English

* fix(telegram): remove 'default' user fallback — resolve user dynamically

- botUserID no longer captured once at startup (was 'default' if no user yet)
- resolveBotUser() reads first registered user from DB on demand:
  * called on every /start (handles: registered after bot launch)
  * called before every AI message (handles mid-session registration)
- If no user registered: clear English error 'No account found. Please register on the web UI first'
- start.sh: fix set_env_var appending without newline (token was concatenated to prev line)

* refactor(telegram): clean onboarding — web UI for setup, Telegram for operations

- /start shows clean status: 'setup required → open web UI' or 'ready → examples'
- Removed tryHandleSetupCommand (no more CLI-style 'configure deepseek sk-xxx')
- Removed automatic language selection on /start (use /lang anytime instead)
- newLLMClient returns nil when no model → clear guard, not fallback
- statusMsg() replaces buildSetupGuide(): two states only (missing config / ready)
- Bot is now purely an operations interface; config lives in the web UI

* refactor: single-user web-based setup — replace env config with Settings UI

Move from multi-user env-var config to single-user web-first architecture:
- Add SetupPage for first-time initialization (replaces /register)
- Add SettingsPage for AI models, exchanges, Telegram, and password management
- Enrich all API route schemas with exact ID usage documentation
- Add PUT /user/password endpoint for in-app password changes
- Remove REGISTRATION_ENABLED, MAX_USERS, TELEGRAM_BOT_TOKEN from env config
- Simplify LoginPage design, remove admin mode and registration links
- Telegram bot now resolves user email for identity display
- start.sh no longer runs interactive Telegram setup

* feat: add blockRun (x402 USDC) support to all AI model consumers

- telegram/bot.go: add blockrun-base, blockrun-sol, minimax to
  clientForProvider; fix newLLMClient to prefer TelegramConfig.ModelID
  over GetDefault; log USDC payment provider usage
- debate/engine.go: add blockrun-base, blockrun-sol to InitializeClients
- api/strategy.go: add blockrun-base, blockrun-sol to runRealAITest
- backtest/ai_client.go: add blockrun-base, blockrun-sol to configureMCPClient

* feat: add Claw402 (claw402.ai) x402 USDC payment provider

Add Claw402Client for claw402.ai's x402 micropayment gateway (Base USDC).
Supports 15+ AI models (GPT-5.4, Claude Opus, DeepSeek, Qwen, Grok, etc.)
with per-model endpoint routing.

- mcp/claw402.go: new client with model→endpoint mapping, x402 v2 payment flow
- mcp/blockrun_base.go: extract shared signX402Payment() for reuse
- Register "claw402" provider in all 6 consumer switch statements:
  api/server.go, api/strategy.go, trader/auto_trader.go,
  telegram/bot.go, debate/engine.go, backtest/ai_client.go

* feat: redesign Claw402 model config UI — friendly wallet setup, USDC guide, official logo, nginx no-cache for index.html

* refactor: centralize x402 payment flow into shared mcp/x402.go

Extract duplicated doRequestWithPayment/call/CallWithRequestFull/buildRequest/
setAuthHeader (~165 lines x3) into shared helpers in mcp/x402.go. Consolidate
shared types (x402v2PaymentRequired, x402AcceptOption, x402Resource) and remove
duplicate Solana types. Fix validAfter to 0 (official SDK standard), drain 402
body before retry, log Payment-Response tx hash, check Payment-Required before
X-Payment-Required.

* fix: stop PR template bot from overwriting user-written descriptions

The pr-template-suggester workflow was triggered on opened/edited/synchronize
events and forcefully replaced the PR body with a template when body < 100 chars.
This caused user-written descriptions to be overwritten.

Replace with a lightweight labeler (OpenClaw-style) that:
- Only adds labels (backend/frontend/docs, size: XS/S/M/L/XL)
- Never modifies the PR body
- Simplified unified PR template at .github/pull_request_template.md

* chore: simplify PR template (OpenClaw-style)
2026-03-11 16:01:42 +08:00

407 lines
10 KiB
Go

// Package store provides unified database storage layer
// All database operations should go through this package
package store
import (
"database/sql"
"fmt"
"nofx/logger"
"sync"
"gorm.io/gorm"
)
// Store unified data storage interface
type Store struct {
gdb *gorm.DB // GORM database connection
db *sql.DB // Legacy sql.DB for backward compatibility
driver *DBDriver // Database driver for abstraction (legacy)
// Sub-stores (lazy initialization)
user *UserStore
aiModel *AIModelStore
exchange *ExchangeStore
trader *TraderStore
decision *DecisionStore
backtest *BacktestStore
position *PositionStore
strategy *StrategyStore
equity *EquityStore
order *OrderStore
grid *GridStore
telegramConfig TelegramConfigStore
mu sync.RWMutex
}
// New creates new Store instance (SQLite mode for backward compatibility)
func New(dbPath string) (*Store, error) {
gdb, err := InitGorm(dbPath)
if err != nil {
return nil, fmt.Errorf("failed to open database: %w", err)
}
// Get underlying sql.DB for legacy compatibility
sqlDB, err := gdb.DB()
if err != nil {
return nil, fmt.Errorf("failed to get sql.DB: %w", err)
}
s := &Store{gdb: gdb, db: sqlDB}
// Initialize all table structures
if err := s.initTables(); err != nil {
sqlDB.Close()
return nil, fmt.Errorf("failed to initialize table structure: %w", err)
}
// Initialize default data
if err := s.initDefaultData(); err != nil {
sqlDB.Close()
return nil, fmt.Errorf("failed to initialize default data: %w", err)
}
logger.Infof("✅ Database initialized (GORM, SQLite)")
return s, nil
}
// NewWithConfig creates new Store instance with provided database configuration
func NewWithConfig(cfg DBConfig) (*Store, error) {
gdb, err := InitGormWithConfig(cfg)
if err != nil {
return nil, fmt.Errorf("failed to open database: %w", err)
}
// Get underlying sql.DB for legacy compatibility
sqlDB, err := gdb.DB()
if err != nil {
return nil, fmt.Errorf("failed to get sql.DB: %w", err)
}
s := &Store{gdb: gdb, db: sqlDB}
// Initialize all table structures
if err := s.initTables(); err != nil {
sqlDB.Close()
return nil, fmt.Errorf("failed to initialize table structure: %w", err)
}
// Initialize default data
if err := s.initDefaultData(); err != nil {
sqlDB.Close()
return nil, fmt.Errorf("failed to initialize default data: %w", err)
}
dbTypeStr := "SQLite"
if cfg.Type == DBTypePostgres {
dbTypeStr = "PostgreSQL"
}
logger.Infof("✅ Database initialized (GORM, %s)", dbTypeStr)
return s, nil
}
// NewFromGorm creates Store from existing GORM connection
func NewFromGorm(gdb *gorm.DB) (*Store, error) {
sqlDB, err := gdb.DB()
if err != nil {
return nil, err
}
return &Store{gdb: gdb, db: sqlDB}, nil
}
// NewFromDB creates Store from existing database connection (legacy)
// Deprecated: Use NewFromGorm instead
func NewFromDB(db *sql.DB) *Store {
return &Store{db: db}
}
// initTables initializes all database tables using GORM AutoMigrate
func (s *Store) initTables() error {
// Create system_config table (GORM handles this via raw SQL for simplicity)
if err := s.gdb.Exec(`
CREATE TABLE IF NOT EXISTS system_config (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
)
`).Error; err != nil {
return fmt.Errorf("failed to create system_config table: %w", err)
}
// Initialize sub-store tables
if err := s.User().initTables(); err != nil {
return fmt.Errorf("failed to initialize user tables: %w", err)
}
if err := s.AIModel().initTables(); err != nil {
return fmt.Errorf("failed to initialize AI model tables: %w", err)
}
if err := s.Exchange().initTables(); err != nil {
return fmt.Errorf("failed to initialize exchange tables: %w", err)
}
if err := s.Trader().initTables(); err != nil {
return fmt.Errorf("failed to initialize trader tables: %w", err)
}
if err := s.Decision().initTables(); err != nil {
return fmt.Errorf("failed to initialize decision log tables: %w", err)
}
if err := s.Backtest().initTables(); err != nil {
return fmt.Errorf("failed to initialize backtest tables: %w", err)
}
if err := s.Position().InitTables(); err != nil {
return fmt.Errorf("failed to initialize position tables: %w", err)
}
if err := s.Strategy().initTables(); err != nil {
return fmt.Errorf("failed to initialize strategy tables: %w", err)
}
if err := s.Equity().initTables(); err != nil {
return fmt.Errorf("failed to initialize equity tables: %w", err)
}
if err := s.Order().InitTables(); err != nil {
return fmt.Errorf("failed to initialize order tables: %w", err)
}
if err := s.Grid().InitTables(); err != nil {
return fmt.Errorf("failed to initialize grid tables: %w", err)
}
if err := s.TelegramConfig().(*telegramConfigStore).initTables(); err != nil {
return fmt.Errorf("failed to initialize telegram config tables: %w", err)
}
return nil
}
// initDefaultData initializes default data
func (s *Store) initDefaultData() error {
if err := s.AIModel().initDefaultData(); err != nil {
return err
}
if err := s.Exchange().initDefaultData(); err != nil {
return err
}
if err := s.Strategy().initDefaultData(); err != nil {
return err
}
// Migrate old decision_account_snapshots data to new trader_equity_snapshots table
if migrated, err := s.Equity().MigrateFromDecision(); err != nil {
logger.Warnf("failed to migrate equity data: %v", err)
} else if migrated > 0 {
logger.Infof("✅ Migrated %d equity records to new table", migrated)
}
return nil
}
// User gets user storage
func (s *Store) User() *UserStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.user == nil {
s.user = NewUserStore(s.gdb)
}
return s.user
}
// AIModel gets AI model storage
func (s *Store) AIModel() *AIModelStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.aiModel == nil {
s.aiModel = NewAIModelStore(s.gdb)
}
return s.aiModel
}
// Exchange gets exchange storage
func (s *Store) Exchange() *ExchangeStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.exchange == nil {
s.exchange = NewExchangeStore(s.gdb)
}
return s.exchange
}
// Trader gets trader storage
func (s *Store) Trader() *TraderStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.trader == nil {
s.trader = NewTraderStore(s.gdb)
}
return s.trader
}
// Decision gets decision log storage
func (s *Store) Decision() *DecisionStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.decision == nil {
s.decision = NewDecisionStore(s.gdb)
}
return s.decision
}
// Backtest gets backtest data storage
func (s *Store) Backtest() *BacktestStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.backtest == nil {
s.backtest = NewBacktestStore(s.gdb)
}
return s.backtest
}
// Position gets position storage
func (s *Store) Position() *PositionStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.position == nil {
s.position = NewPositionStore(s.gdb)
}
return s.position
}
// Strategy gets strategy storage
func (s *Store) Strategy() *StrategyStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.strategy == nil {
s.strategy = NewStrategyStore(s.gdb)
}
return s.strategy
}
// Equity gets equity storage
func (s *Store) Equity() *EquityStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.equity == nil {
s.equity = NewEquityStore(s.gdb)
}
return s.equity
}
// Order gets order storage
func (s *Store) Order() *OrderStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.order == nil {
s.order = NewOrderStore(s.gdb)
}
return s.order
}
// Grid gets grid trading storage
func (s *Store) Grid() *GridStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.grid == nil {
s.grid = NewGridStore(s.gdb)
}
return s.grid
}
// TelegramConfig gets Telegram bot configuration storage
func (s *Store) TelegramConfig() TelegramConfigStore {
s.mu.Lock()
defer s.mu.Unlock()
if s.telegramConfig == nil {
s.telegramConfig = NewTelegramConfigStore(s.gdb)
}
return s.telegramConfig
}
// Close closes database connection
func (s *Store) Close() error {
if s.driver != nil {
return s.driver.Close()
}
if s.db != nil {
return s.db.Close()
}
return nil
}
// GormDB returns the GORM database connection
func (s *Store) GormDB() *gorm.DB {
return s.gdb
}
// Driver returns database driver for abstraction (legacy)
func (s *Store) Driver() *DBDriver {
return s.driver
}
// DBType returns current database type
func (s *Store) DBType() DBType {
if s.driver != nil {
return s.driver.Type
}
// Detect from GORM dialector
if s.gdb != nil {
switch s.gdb.Dialector.Name() {
case "postgres":
return DBTypePostgres
default:
return DBTypeSQLite
}
}
return DBTypeSQLite
}
// q converts query placeholders for current database type (legacy helper)
func (s *Store) q(query string) string {
return convertQuery(query, s.DBType())
}
// DB gets underlying database connection (for legacy code compatibility)
// Deprecated: use GormDB() instead
func (s *Store) DB() *sql.DB {
return s.db
}
// GetSystemConfig gets a system configuration value by key
func (s *Store) GetSystemConfig(key string) (string, error) {
var value string
result := s.gdb.Raw("SELECT value FROM system_config WHERE key = ?", key).Scan(&value)
if result.Error != nil {
if result.Error == gorm.ErrRecordNotFound {
return "", nil
}
return "", result.Error
}
if result.RowsAffected == 0 {
return "", nil
}
return value, nil
}
// SetSystemConfig sets a system configuration value
func (s *Store) SetSystemConfig(key, value string) error {
// Use GORM-compatible upsert
return s.gdb.Exec(`
INSERT INTO system_config (key, value) VALUES (?, ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value
`, key, value).Error
}
// Transaction executes transaction with GORM
func (s *Store) Transaction(fn func(tx *gorm.DB) error) error {
return s.gdb.Transaction(fn)
}
// TransactionSQL executes transaction with sql.Tx (legacy)
// Deprecated: Use Transaction() instead
func (s *Store) TransactionSQL(fn func(tx *sql.Tx) error) error {
tx, err := s.db.Begin()
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
}
if err := fn(tx); err != nil {
tx.Rollback()
return err
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("failed to commit transaction: %w", err)
}
return nil
}