* feat(telegram): add AI agent bot with streaming and account context
- Add Telegram bot with long-polling and AI agent loop (api_call tool)
- SSE streaming with real-time message editing and ⏳ placeholder
- Account state injection at conversation start (models, exchanges,
strategies, traders, per-trader PnL and statistics)
- Lane semaphore per chat serializes concurrent messages (60s timeout)
- Idle timeout watchdog (60s) prevents hung streaming connections
- Look-ahead buffer prevents partial <api_call> tag leaking to user
- Fix PUT /strategies/:id to merge config (read-then-merge pattern)
- Add route registry with full API schema for LLM documentation
- Add TelegramConfig store and Web UI config modal
- Add GetAnyEnabled to AIModel store for bot LLM client selection
* fix(telegram): eliminate narration, add full-setup workflow and tests
- Rewrite NO NARRATION rule: response is EITHER api_call tag alone OR
final text reply — no text before api_call under any circumstances
- Ban all narration patterns: 现在我将/好的/正在/I will/Let me etc.
- Add 'create strategy + create trader + start' full setup workflow
- Add 12 automated tests covering:
- No narration leaking to user (5 narration variants tested)
- api_call tag never leaks to user
- Full setup workflow: POST strategy → verify → POST trader → start
- Start existing trader workflow
- Max iterations safety, tag stripping, parser edge cases
* refactor(agent): replace XML api_call with native function calling
Migrate the Telegram bot agent from an XML tag hack (<api_call>) to
OpenAI-native function calling via CallWithRequestFull.
Key changes:
- mcp/interface.go: add parseMCPResponseFull to clientHooks interface
- mcp/client.go: route callWithRequestFull through hooks for overridability
- mcp/claude_client.go: override parseMCPResponseFull for Claude response
format (tool_use blocks instead of choices[].message.tool_calls)
- telegram/agent/agent.go: rewrite Run() to use CallWithRequestFull;
define api_request tool with JSON Schema; implement tool-call loop
with role="tool" result messages; remove XML parsing entirely
- telegram/agent/apicall.go: remove parseAPICall (dead code)
- telegram/agent/prompt.go: simplify — remove XML format instructions,
replace with concise api_request tool usage instructions
- telegram/agent/agent_test.go: rebuild all tests using LLMResponse
objects; add TestNarrationStructurallyImpossible, TestOnChunkCalledWithFinalReply,
TestToolCallIDPropagated; remove XML-specific tests
Architecture advantage: with native function calling, the LLM returns
EITHER ToolCalls OR Content — never both. Narration is now structurally
impossible at the protocol level, not just enforced by prompt rules.
All 11 agent tests pass. mcp package tests pass.
* refactor(mcp): route buildRequestBodyFromRequest through hooks + full Anthropic format
Problem: callWithRequest/Full/Stream all called client.buildRequestBodyFromRequest
directly (not via hooks), so ClaudeClient could never override it. This meant
tool calling sent OpenAI format to Anthropic (wrong field names, wrong roles).
Changes:
mcp/interface.go
- Add buildRequestBodyFromRequest(*Request) map[string]any to clientHooks
- Improve comments: document what each hook group does and why
mcp/client.go
- All three paths (callWithRequest, callWithRequestFull, CallWithRequestStream)
now call client.hooks.buildRequestBodyFromRequest — ClaudeClient picks up
mcp/claude_client.go
- Full rewrite with format comparison table in package doc
- buildRequestBodyFromRequest: produces correct Anthropic wire format
* system prompt → top-level "system" field
* tools: parameters → input_schema, no "type:function" wrapper
* tool_choice "auto" → {"type":"auto"} object
* assistant tool calls → content[{type:tool_use, id, name, input}]
* role=tool results → role=user content[{type:tool_result,...}]
* consecutive tool results merged into single user turn
- convertMessagesToAnthropic: handles all three message types
- parseMCPResponseFull: extracts text + tool_use blocks
- parseMCPResponse: delegates to parseMCPResponseFull
All mcp and agent tests pass.
* fix(telegram): fix claude client dispatch + strategy creation workflow
- telegram/bot.go: clientForProvider now returns NewClaudeClient() for
'claude' provider (was incorrectly falling back to DeepSeekClient which
uses OpenAI wire format, breaking Anthropic API calls)
- api/server.go: fix scan_interval_minutes schema default (3, not 60);
POST /api/strategies now clearly states config is OPTIONAL with complete
working defaults; POST /api/traders removes redundant GET workflow note
- telegram/agent/prompt.go: simplify strategy creation — just POST {name}
without config (backend applies full working defaults automatically);
only include config when user requests custom settings
* test(mcp): add ClaudeClient wire format tests
Tests cover all Anthropic-specific format conversions:
- system prompt lifted to top-level field
- tools use input_schema (not parameters)
- tool_choice is object {type:auto} not string
- assistant tool calls → content[{type:tool_use}]
- consecutive tool results merged into single user turn
- parseMCPResponseFull: text, tool_use, and error cases
- x-api-key header (not Authorization: Bearer)
- /messages endpoint URL
* fix(telegram): clientForProvider returns correct client for all 7 providers
Previously qwen/kimi/grok/gemini all fell back to DeepSeekClient.
Each provider now gets its own dedicated client with correct default
base URL and model. All 7 providers now fully supported:
openai, deepseek, claude, qwen, kimi, grok, gemini
* fix(telegram): newLLMClient uses bound user's model, not any user's model
GetAnyEnabled() searched across all users in DB — if user B has an
enabled model, bot could use their API key while acting as user A.
Now uses GetDefault(botUserID) which only looks up the bound user's
enabled model, matching the same user scope as all API calls.
* fix(auth): single-user deployment by default, no open registration
Registration logic redesigned:
- Empty DB (first-time setup): registration always open, no config needed
- After first user exists: registration closed by default
- Multi-user opt-in: set REGISTRATION_ENABLED=true + MAX_USERS=N in .env
Config defaults changed:
- RegistrationEnabled: true → false (closed after first user)
- MaxUsers: 10 → 1 (single-user deployment default)
This eliminates the confusion of multiple users appearing in a personal
deployment where Telegram is bound to a single admin account.
* feat(solo): beginner-friendly onboarding — smart setup guide + direct config commands
start.sh:
- Interactive Telegram Bot Token prompt on first run
- Token format validation (must match 12345:ABC... pattern)
- Friendly step-by-step startup instructions after launch
telegram/bot.go:
- /start now shows context-aware setup guide based on actual config state:
- No AI model → explains how to configure, lists all providers
- AI model OK but no exchange → guides to configure exchange via chat
- All configured → full capabilities welcome message
- New: direct setup commands ('配置 deepseek sk-xxx') bypass LLM entirely
so AI model can be configured even before any model exists (bootstrap fix)
- All messages now in Chinese (匹配用户语言)
telegram/agent/prompt.go:
- Added first-time setup detection section
- Agent told to never ask user to visit web UI — everything via chat
* feat(i18n): bilingual EN/ZH setup guide with language selection
store/telegram_config.go:
- Add Language field to TelegramConfig (persisted in DB)
- Add SetLanguage(lang) and GetLanguage() methods
- Default language: English (en)
telegram/bot.go:
- First /start triggers language selection (1=English, 2=中文)
- /lang command to change language at any time
- awaitingLang state machine handles language choice before any other input
- buildSetupGuide() now fully bilingual (EN/ZH), context-aware:
Step 1: configure AI model (no model yet)
Step 2: configure exchange (model OK, no exchange)
Ready: show full capabilities
- tryHandleSetupCommand() bilingual: 'configure/配置 <provider> <key>'
- helpMessage(lang) fully bilingual
- All error/status messages bilingual
Default: English. isLangDefault() detects whether user has explicitly
chosen a language vs falling back to the 'en' default.
* fix(telegram): use Markdown rendering + simplify language selection condition
- sendMarkdownMsg() helper: sends with ParseMode=Markdown, falls back to plain text
- All formatted messages (langSelectionMsg, buildSetupGuide, helpMessage) now render
bold text and code blocks correctly in Telegram
- Simplify /start language check: isLangDefault(st) alone is sufficient
(lang == 'en' && isLangDefault was redundant — GetLanguage returns 'en' when empty)
* fix(start.sh): translate all user-facing text to English
Entire script was in Chinese. Now English-first throughout:
- startup banner, prompts, success/error messages
- setup_telegram(): English instructions and validation messages
- start(): English next-steps after launch
- stop/restart/clean/update/regenerate-keys/show_help: all English
* fix(telegram): remove 'default' user fallback — resolve user dynamically
- botUserID no longer captured once at startup (was 'default' if no user yet)
- resolveBotUser() reads first registered user from DB on demand:
* called on every /start (handles: registered after bot launch)
* called before every AI message (handles mid-session registration)
- If no user registered: clear English error 'No account found. Please register on the web UI first'
- start.sh: fix set_env_var appending without newline (token was concatenated to prev line)
* refactor(telegram): clean onboarding — web UI for setup, Telegram for operations
- /start shows clean status: 'setup required → open web UI' or 'ready → examples'
- Removed tryHandleSetupCommand (no more CLI-style 'configure deepseek sk-xxx')
- Removed automatic language selection on /start (use /lang anytime instead)
- newLLMClient returns nil when no model → clear guard, not fallback
- statusMsg() replaces buildSetupGuide(): two states only (missing config / ready)
- Bot is now purely an operations interface; config lives in the web UI
* refactor: single-user web-based setup — replace env config with Settings UI
Move from multi-user env-var config to single-user web-first architecture:
- Add SetupPage for first-time initialization (replaces /register)
- Add SettingsPage for AI models, exchanges, Telegram, and password management
- Enrich all API route schemas with exact ID usage documentation
- Add PUT /user/password endpoint for in-app password changes
- Remove REGISTRATION_ENABLED, MAX_USERS, TELEGRAM_BOT_TOKEN from env config
- Simplify LoginPage design, remove admin mode and registration links
- Telegram bot now resolves user email for identity display
- start.sh no longer runs interactive Telegram setup
* feat: add blockRun (x402 USDC) support to all AI model consumers
- telegram/bot.go: add blockrun-base, blockrun-sol, minimax to
clientForProvider; fix newLLMClient to prefer TelegramConfig.ModelID
over GetDefault; log USDC payment provider usage
- debate/engine.go: add blockrun-base, blockrun-sol to InitializeClients
- api/strategy.go: add blockrun-base, blockrun-sol to runRealAITest
- backtest/ai_client.go: add blockrun-base, blockrun-sol to configureMCPClient
* feat: add Claw402 (claw402.ai) x402 USDC payment provider
Add Claw402Client for claw402.ai's x402 micropayment gateway (Base USDC).
Supports 15+ AI models (GPT-5.4, Claude Opus, DeepSeek, Qwen, Grok, etc.)
with per-model endpoint routing.
- mcp/claw402.go: new client with model→endpoint mapping, x402 v2 payment flow
- mcp/blockrun_base.go: extract shared signX402Payment() for reuse
- Register "claw402" provider in all 6 consumer switch statements:
api/server.go, api/strategy.go, trader/auto_trader.go,
telegram/bot.go, debate/engine.go, backtest/ai_client.go
* feat: redesign Claw402 model config UI — friendly wallet setup, USDC guide, official logo, nginx no-cache for index.html
* refactor: centralize x402 payment flow into shared mcp/x402.go
Extract duplicated doRequestWithPayment/call/CallWithRequestFull/buildRequest/
setAuthHeader (~165 lines x3) into shared helpers in mcp/x402.go. Consolidate
shared types (x402v2PaymentRequired, x402AcceptOption, x402Resource) and remove
duplicate Solana types. Fix validAfter to 0 (official SDK standard), drain 402
body before retry, log Payment-Response tx hash, check Payment-Required before
X-Payment-Required.
* fix: stop PR template bot from overwriting user-written descriptions
The pr-template-suggester workflow was triggered on opened/edited/synchronize
events and forcefully replaced the PR body with a template when body < 100 chars.
This caused user-written descriptions to be overwritten.
Replace with a lightweight labeler (OpenClaw-style) that:
- Only adds labels (backend/frontend/docs, size: XS/S/M/L/XL)
- Never modifies the PR body
- Simplified unified PR template at .github/pull_request_template.md
* chore: simplify PR template (OpenClaw-style)
- Migrate all store packages from raw database/sql to GORM ORM
- Add PostgreSQL support alongside SQLite
- Move EncryptedString type to crypto package for cleaner architecture
- Add automatic encryption/decryption for sensitive fields (API keys, secrets)
- Fix PostgreSQL AutoMigrate conflicts by skipping existing tables
- Fix duplicate /klines route registration
- Update tests to use GORM database connections
- Add database configuration support in config package
- Fix xyz dex balance calculation (use marginSummary for isolated margin)
- Add Alpaca provider for US stocks market data
- Add TwelveData provider for forex & metals market data
- Add Hyperliquid kline provider
- Centralize API keys in config system
- Add builder fee for order routing
- Improve chart UI with compact design
- Fix position history fee display precision
- Add comprehensive balance calculation tests
- Change default MaxUsers from 5 to 10 in config
- Add update reminder section in install.sh output
- Add 'Keeping Updated' section in all README translations
- Add real-time position display with unrealized P&L during backtest
- Fix P&L calculation by tracking accumulated opening fees
- Add strategy coin source resolution (AI500, OI Top, mixed)
- Infer AI provider from model name for better compatibility
- Cap position size to available margin to prevent insufficient cash errors
- Fix trade markers on K-line chart (long/short instead of buy/sell)
- Add QuantData and OI ranking to backtest decision context
- Add TRANSPORT_ENCRYPTION env config (default: false)
- Allow HTTP/IP access when transport encryption is disabled
- Add /api/crypto/config endpoint to expose encryption status
- Update WebCryptoEnvironmentCheck with 'disabled' status
- Update ExchangeConfigModal and AITradersPage to allow form submission when disabled
- Add i18n translations for disabled status (EN/CN)
- Update README with two deployment modes documentation
- Remove system_config, beta_codes, signal_source tables and related code
- Simplify config.go to only read from .env (APIServerPort, JWTSecret, RegistrationEnabled)
- Remove GetCustomCoins, use all USDT perpetual contracts for WSMonitor
- Add trader_equity_snapshots table for equity tracking
- Remove signal source modal from frontend AITradersPage
- Fix WSMonitor nil panic by restoring initialization in main.go
* refactor: 简化交易动作，移除 update_stop_loss/update_take_profit/partial_close
- 移除 Decision 结构体中的 NewStopLoss, NewTakeProfit, ClosePercentage 字段
- 删除 executeUpdateStopLossWithRecord, executeUpdateTakeProfitWithRecord, executePartialCloseWithRecord 函数
- 简化 logger 中的 partial_close 聚合逻辑
- 更新 AI prompt 和验证逻辑，只保留 6 个核心动作
- 清理相关测试代码
保留的交易动作: open_long, open_short, close_long, close_short, hold, wait
* refactor: 移除 AI学习与反思 模块
- 删除前端 AILearning.tsx 组件和相关引用
- 删除后端 /performance API 接口
- 删除 logger 中 AnalyzePerformance、calculateSharpeRatio 等函数
- 删除 PerformanceAnalysis、TradeOutcome、SymbolPerformance 等结构体
- 删除 Context 中的 Performance 字段
- 移除 AI prompt 中夏普比率自我进化相关内容
- 清理 i18n 翻译文件中的相关条目
该模块基于磁盘存储计算，经常出错，做减法移除
* refactor: 将数据库操作统一迁移到 store 包
- 新增 store/ 包，统一管理所有数据库操作
- store.go: 主 Store 结构，懒加载各子模块
- user.go, ai_model.go, exchange.go, trader.go 等子模块
- 支持加密/解密函数注入 (SetCryptoFuncs)
- 更新 main.go 使用 store.New() 替代 config.NewDatabase()
- 更新 api/server.go 使用 *store.Store 替代 *config.Database
- 更新 manager/trader_manager.go:
- 新增 LoadTradersFromStore, LoadUserTradersFromStore 方法
- 删除旧版 LoadUserTraders, LoadTraderByID, loadSingleTrader 等方法
- 移除 nofx/config 依赖
- 删除 config/database.go 和 config/database_test.go
- 更新 api/server_test.go 使用 store.Trader 类型
- 清理 logger/ 包中未使用的 telegram 相关代码
* refactor: unify encryption key management via .env
- Remove redundant EncryptionManager and SecureStorage
- Simplify CryptoService to load keys from environment variables only
- RSA_PRIVATE_KEY: RSA private key for client-server encryption
- DATA_ENCRYPTION_KEY: AES-256 key for database encryption
- JWT_SECRET: JWT signing key for authentication
- Update start.sh to auto-generate missing keys on first run
- Remove secrets/ directory and file-based key storage
- Delete obsolete encryption setup scripts
- Update .env.example with all required keys
* refactor: unify logger usage across mcp package
- Add MCPLogger adapter in logger package to implement mcp.Logger interface
- Update mcp/config.go to use global logger by default
- Remove redundant defaultLogger from mcp/logger.go
- Keep noopLogger for testing purposes
* chore: remove leftover test RSA key file
* chore: remove unused bootstrap package
* refactor: unify logging to use logger package instead of fmt/log
- Replace all fmt.Print/log.Print calls with logger package
- Add auto-initialization in logger package init() for test compatibility
- Update main.go to initialize logger at startup
- Migrate all packages: api, backtest, config, decision, manager, market, store, trader
* refactor: rename database file from config.db to data.db
- Update main.go, start.sh, docker-compose.yml
- Update migration script and documentation
- Update .gitignore and translations
* fix: add RSA_PRIVATE_KEY to docker-compose environment
* fix: add registration_enabled to /api/config response
* fix: Fix navigation between login and register pages
Use window.location.href instead of react-router's navigate() to fix
the issue where URL changes but the page doesn't reload due to App.tsx
using custom route state management.
* fix: Switch SQLite from WAL to DELETE mode for Docker compatibility
WAL mode causes data sync issues with Docker bind mounts on macOS due
to incompatible file locking mechanisms between the container and host.
DELETE mode (traditional journaling) ensures data is written directly
to the main database file.
* refactor: Remove default user from database initialization
The default user was a legacy placeholder that is no longer needed now
that proper user registration is in place.
* feat: Add order tracking system with centralized status sync
- Add trader_orders table for tracking all order lifecycle
- Implement GetOrderStatus interface for all exchanges (Binance, Bybit, Hyperliquid, Aster, Lighter)
- Create OrderSyncManager for centralized order status polling
- Add trading statistics (Sharpe ratio, win rate, profit factor) to AI context
- Include recent completed orders in AI decision input
- Remove per-order goroutine polling in favor of global sync manager
* feat: Add TradingView K-line chart to dashboard
- Create TradingViewChart component with exchange/symbol selectors
- Support Binance, Bybit, OKX, Coinbase, Kraken, KuCoin exchanges
- Add popular symbols quick selection
- Support multiple timeframes (1m to 1W)
- Add fullscreen mode
- Integrate with Dashboard page below equity chart
- Add i18n translations for zh/en
* refactor: Replace separate charts with tabbed ChartTabs component
- Create ChartTabs component with tab switching between equity curve and K-line
- Add embedded mode support for EquityChart and TradingViewChart
- User can now switch between account equity and market chart in same area
* fix: Use ChartTabs in App.tsx and fix embedded mode in EquityChart
- Replace EquityChart with ChartTabs in App.tsx (the actual dashboard renderer)
- Fix EquityChart embedded mode for error and empty data states
- Rename interval state to timeInterval to avoid shadowing window.setInterval
- Add debug logging to ChartTabs component
* feat: Add position tracking system for accurate trade history
- Add trader_positions table to track complete open/close trades
- Add PositionSyncManager to detect manual closes via polling
- Record position on open, update on close with PnL calculation
- Use positions table for trading stats and recent trades (replacing orders table)
- Fix TradingView chart symbol format (add .P suffix for futures)
- Fix DecisionCard wait/hold action color (gray instead of red)
- Auto-append USDT suffix for custom symbol input
* update
---------
* fix: resolve multiple bugs preventing trader creation
Bug fixes:
1. Fix time.Time scanning error - SQLite stores datetime as TEXT, now parsing manually
2. Fix foreign key mismatch - traders table referenced exchanges(id) but exchanges uses composite primary key (id, user_id)
3. Add missing backtestManager field to Server struct
4. Add missing Shutdown method to Server struct
5. Fix NewFuturesTrader call - pass userId parameter
6. Fix UpdateExchange call - pass all required parameters
7. Add migrateTradersTable() to fix existing databases
These issues prevented creating new traders with 500 errors.
* fix(api): fix balance extraction field name mismatch
Binance API returns 'availableBalance' (camelCase) but code was looking for
'available_balance' (snake_case). Now supports both formats.
Also added 'totalWalletBalance' as fallback for total balance extraction.
* fix(frontend): add missing ConfirmDialogProvider to App
The delete trader button required ConfirmDialogProvider to be wrapped
around the App component for the confirmation dialog to work.
---------
Co-authored-by: NOFX Trader <nofx@local.dev>
* refactor(frontend): extract RegistrationDisabled as reusable component
- Create RegistrationDisabled component with i18n support
- Add registrationClosed and registrationClosedMessage translations
- Replace inline JSX in App.tsx with new component
- Improve code maintainability and reusability
- Add hover effect to back button for better UX
* fix(frontend): add registration toggle to LoginModal component
- Add useSystemConfig hook to LoginModal
- Conditionally render registration button based on registration_enabled config
- Ensures consistency with HeaderBar and LoginPage registration controls
- Completes registration toggle feature implementation across all entry points
* feat(frontend): add registration toggle UI support
- Add registration disabled page in App.tsx when registration is closed
- Hide registration link in LoginPage when registration is disabled
- Add registration_enabled field to SystemConfig interface
- Frontend conditionally shows/hides registration UI based on backend config
* feat: add registration toggle feature
Add system-level registration enable/disable control:
- Add registration_enabled config to system_config table (default: true)
- Add registration check in handleRegister API endpoint
- Expose registration_enabled status in /api/config endpoint
- Frontend can use this config to conditionally show/hide registration UI
This allows administrators to control user registration without code changes.
* fix(frontend): add registration toggle to HeaderBar and RegisterPage
- Add useSystemConfig hook and registrationEnabled check to HeaderBar
- Conditionally show/hide signup buttons in both desktop and mobile views
- Add registration check to RegisterPage to show RegistrationDisabled component
- This completes the registration toggle feature across all UI components
* test(frontend): add comprehensive unit tests for registration toggle feature
- Add RegistrationDisabled component tests (rendering, navigation, styling)
- Add registrationToggle logic tests (config handling, edge cases, multi-location consistency)
- Configure Vitest with jsdom environment for React component testing
- All 80 tests passing (9 new tests for RegistrationDisabled + 21 for toggle logic)
* fix(ci): add test encryption key for CI environment
- Add DATA_ENCRYPTION_KEY environment variable to PR test workflow
- Add test RSA public key for encryption tests in CI
- Ensures unit tests pass in CI without production credentials
Co-authored-by: tinkle-community <tinklefund@gmail.com>
* fix(ci): install Go cover tool to eliminate covdata warnings
- Add step to install golang.org/x/tools/cmd/cover in CI workflow
- Use || true to prevent installation failure from breaking CI
- Eliminates "no such tool covdata" warnings during test execution
- Apply go fmt to multiple files for consistency
Co-authored-by: tinkle-community <tinklefund@gmail.com>
* fix(ci): install covdata tool for Go 1.23 coverage
The CI was failing with "go: no such tool 'covdata'" error.
This is because Go 1.23 requires the covdata tool to be installed
for coverage reporting.
Changes:
- Install golang.org/x/tools/cmd/covdata in CI workflow
- Update step name to reflect both coverage tools being installed
Fixes the unit test failures in CI pipeline.
Co-Authored-By: tinkle-community <tinklefund@gmail.com>
* fix(ci): remove unnecessary covdata installation and use builtin go tool cover
The previous attempt to install golang.org/x/tools/cmd/covdata was failing
because the package structure changed in Go 1.23 and tools v0.38.0.
The covdata tool is not needed for this project since we only use simple
coverage reporting with go test -coverprofile. The go tool cover command
is built into the Go toolchain and requires no additional installation.
Changes:
- Remove failed covdata and cover installation attempts
- Add verification step for go tool cover availability
- Simplify CI pipeline by eliminating unnecessary dependencies
Co-authored-by: tinkle-community <tinklefund@gmail.com>
* fix(ci): upgrade Go version to 1.25 to match go.mod declaration
The CI was using Go 1.23 while go.mod declares go 1.25.0, causing
"no such tool covdata" errors during coverage test compilation.
Go 1.25's coverage infrastructure requires toolchain features not
available in Go 1.23.
This change aligns the CI Go version with the project's declared
version requirement, ensuring the full Go 1.25 toolchain (including
the covdata tool) is available for coverage testing.
Co-authored-by: tinkle-community <tinklefund@gmail.com>
---------
Co-authored-by: tinkle-community <tinklefund@gmail.com>
* fix(database): prevent data loss on Docker restart with WAL mode and graceful shutdown
Fixes#816
## Problem
Exchange API keys and private keys were being lost after `docker compose restart`.
This P0 bug posed critical security and operational risks.
### Root Cause
1. **SQLite journal_mode=delete**: Traditional rollback journal doesn't protect
against data loss during non-graceful shutdowns
2. **Incomplete graceful shutdown**: Application relied on `defer database.Close()`
which may not execute before process termination
3. **Docker grace period**: Default 10s may not be sufficient for cleanup
### Data Loss Scenario
```
User updates exchange config → Backend writes to SQLite → Data in buffer (not fsynced)
→ Docker restart (SIGTERM) → App exits → SQLite never flushes → Data lost
```
## Solution
### 1. Enable WAL Mode (Primary Fix)
- **Before**: `journal_mode=delete` (rollback journal)
- **After**: `journal_mode=WAL` (Write-Ahead Logging)
**Benefits:**
- ✅ Crash-safe even during power loss
- ✅ Better concurrent write performance
- ✅ Atomic commits with durability guarantees
### 2. Improve Graceful Shutdown
**Before:**
```go
<-sigChan
traderManager.StopAll()
// defer database.Close() may not execute in time
```
**After:**
```go
<-sigChan
traderManager.StopAll() // Step 1: Stop traders
server.Shutdown() // Step 2: Stop HTTP server (new)
database.Close() // Step 3: Explicit database close (new)
```
### 3. Increase Docker Grace Period
```yaml
stop_grace_period: 30s # Allow 30s for graceful shutdown
```
## Changes
### config/database.go
- Enable `PRAGMA journal_mode=WAL` on database initialization
- Set `PRAGMA synchronous=FULL` for data durability
- Add log message confirming WAL mode activation
### api/server.go
- Add `httpServer *http.Server` field to Server struct
- Implement `Shutdown()` method with 5s timeout
- Replace `router.Run()` with `httpServer.ListenAndServe()` for graceful shutdown support
- Add `context` import for shutdown context
### main.go
- Add explicit shutdown sequence:
1. Stop all traders
2. Shutdown HTTP server (new)
3. Close database connection (new)
- Add detailed logging for each shutdown step
### docker-compose.yml
- Add `stop_grace_period: 30s` to backend service
### config/database_test.go (TDD)
- `TestWALModeEnabled`: Verify WAL mode is active
- `TestSynchronousMode`: Verify synchronous=FULL setting
- `TestDataPersistenceAcrossReopen`: Simulate Docker restart scenario
- `TestConcurrentWritesWithWAL`: Verify concurrent write handling
## Test Results
```bash
$ go test -v ./config
=== RUN TestWALModeEnabled
--- PASS: TestWALModeEnabled (0.25s)
=== RUN TestSynchronousMode
--- PASS: TestSynchronousMode (0.06s)
=== RUN TestDataPersistenceAcrossReopen
--- PASS: TestDataPersistenceAcrossReopen (0.05s)
=== RUN TestConcurrentWritesWithWAL
--- PASS: TestConcurrentWritesWithWAL (0.09s)
PASS
```
All 16 tests pass (including 9 existing + 4 new WAL tests + 3 concurrent tests).
## Impact
**Before:**
- 🔴 Exchange credentials lost on restart
- 🔴 Trading operations disrupted
- 🔴 Security risk from credential re-entry
**After:**
- ✅ Data persistence guaranteed
- ✅ No credential loss after restart
- ✅ Safe graceful shutdown in all scenarios
- ✅ Better concurrent performance
## Acceptance Criteria
- [x] WAL mode enabled in database initialization
- [x] Graceful shutdown explicitly closes database
- [x] Unit tests verify data persistence across restarts
- [x] Docker grace period increased to 30s
- [x] All tests pass
## Deployment Notes
After deploying this fix:
1. Rebuild Docker image: `./start.sh start --build`
2. Existing `config.db` will be automatically converted to WAL mode
3. WAL files (`config.db-wal`, `config.db-shm`) will be created
4. No manual intervention required
## References
- SQLite WAL Mode: https://www.sqlite.org/wal.html
- Go http.Server Graceful Shutdown: https://pkg.go.dev/net/http#Server.Shutdown
* Add config.db* to gitignore
* fix(database): prevent empty values from overwriting exchange private keys
Fixes#781
## Problem
- Empty values were overwriting existing private keys during exchange config updates
- INSERT operations were storing plaintext instead of encrypted values
- Caused data loss when users edited exchange configurations via web UI
## Solution
1. **Dynamic UPDATE**: Only update sensitive fields (api_key, secret_key, aster_private_key) when non-empty
2. **Encrypted INSERT**: Use encrypted values for all sensitive fields during INSERT
3. **Comprehensive tests**: Added 9 unit tests with 90.2% coverage
## Changes
- config/database.go (UpdateExchange): Refactored to use dynamic SQL building
- config/database_test.go (new): Added comprehensive test suite
## Test Results
✅ All 9 tests pass
✅ Coverage: 90.2% of UpdateExchange function (100% of normal paths)
✅ Verified empty values no longer overwrite existing keys
✅ Verified INSERT uses encrypted storage
## Impact
- 🔒 Protects user's exchange API keys and private keys from accidental deletion
- 🔒 Ensures all sensitive data is encrypted at rest
- ✅ Backward compatible: non-empty updates work as before
* revert: remove incorrect INSERT encryption fix - out of scope
## Background
Hyperliquid official documentation recommends using Agent Wallet pattern for API trading:
- Agent Wallet is used for signing only
- Main Wallet Address is used for querying account data
- Agent Wallet should not hold significant funds
Reference: https://hyperliquid.gitbook.io/hyperliquid-docs/for-developers/api/nonces-and-api-wallets
## Current Implementation
Current implementation allows auto-generating wallet address from private key,
which simplifies user configuration but may lead to potential security concerns
if users accidentally use their main wallet private key.
## Enhancement
Following the proven pattern already used in Aster exchange implementation
(which uses dual-address mode), this enhancement upgrades Hyperliquid to
Agent Wallet mode:
### Core Changes
1. **Mandatory dual-address configuration**
- Agent Private Key (for signing)
- Main Wallet Address (holds funds)
2. **Multi-layer security checks**
- Detect if user accidentally uses main wallet private key
- Validate Agent wallet balance (reject if > 100 USDC)
- Provide detailed configuration guidance
3. **Design consistency**
- Align with Aster's dual-address pattern
- Follow Hyperliquid official best practices
### Code Changes
**config/database.go**:
- Add inline comments clarifying Agent Wallet security model
**trader/hyperliquid_trader.go**:
- Require explicit main wallet address (no auto-generation)
- Check if agent address matches main wallet address (security risk indicator)
- Query agent wallet balance and block if excessive
- Display both agent and main wallet addresses for transparency
**web/src/components/AITradersPage.tsx**:
- Add security alert banner explaining Agent Wallet mode
- Separate required inputs for Agent Private Key and Main Wallet Address
- Add field descriptions and validation
### Benefits
- ✅ Aligns with Hyperliquid official security recommendations
- ✅ Maintains design consistency with Aster implementation
- ✅ Multi-layer protection against configuration mistakes
- ✅ Detailed logging for troubleshooting
### Breaking Change
Users must now explicitly provide main wallet address (hyperliquid_wallet_addr).
Old configurations will receive clear error messages with migration guidance.
### Migration Guide
**Before** (single private key):
```json
{
"hyperliquid_private_key": "0x..."
}
```
**After** (Agent Wallet mode):
```json
{
"hyperliquid_private_key": "0x...", // Agent Wallet private key
"hyperliquid_wallet_addr": "0x..." // Main Wallet address
}
```
Users can create Agent Wallet on Hyperliquid official website:
https://app.hyperliquid.xyz/ → Settings → API Wallets
Co-authored-by: tinkle-community <tinklefund@gmail.com>
- Add soft delete support for AI models
* Add deleted field to ai_models table
* Implement soft delete with sensitive data cleanup
* Filter deleted records in queries
- Replace browser confirm dialogs with custom styled modals
* Create DeleteConfirmModal component with Binance theme
* Add proper warning messages and icons
* Improve user experience with consistent styling
- Fix duplicate model/exchange display in selection dropdowns
* Use supportedModels/supportedExchanges for modal selectors
* Use configuredModels/configuredExchanges for panel display
* Remove redundant selectableModels/selectableExchanges logic
- Enhance data management
* Refresh data after deletion operations
* Proper state cleanup after modal operations
* Clear sensitive data during soft delete
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: tinkle-community <tinklefund@gmail.com>
- Add PostgreSQL + SQLite hybrid database support with automatic switching
- Implement frontend AES-GCM + RSA-OAEP encryption for sensitive data
- Add comprehensive DatabaseInterface with all required methods
- Fix compilation issues with interface consistency
- Update all database method signatures to use DatabaseInterface
- Add missing UpdateTraderInitialBalance method to PostgreSQL implementation
- Integrate RSA public key distribution via /api/config endpoint
- Add frontend crypto service with proper error handling
- Support graceful degradation between encrypted and plaintext transmission
- Add directory creation for RSA keys and PEM parsing fixes
- Test both SQLite and PostgreSQL modes successfully
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: tinkle-community <tinklefund@gmail.com>
## Summary
- Add POST /traders/:id/sync-balance endpoint (Option B)
- Add smart detection showing balance change percentage (Option C)
- Fix balance display bug caused by commit 2b9c4d2
## Changes
### api/server.go
- Add handleSyncBalance() handler
- Query actual exchange balance via trader.GetBalance()
- Calculate change percentage for smart detection
- Update initial_balance in database
- Reload trader into memory after update
### config/database.go
- Add UpdateTraderInitialBalance() method
- Update traders.initial_balance field
## Root Cause
Commit 2b9c4d2 auto-queries exchange balance at trader creation time,
but never updates after user deposits more funds, causing:
- Wrong initial_balance (400 USDT vs actual 3000 USDT)
- Wrong P&L calculations (-2598.55 USDT instead of actual)
## Solution
Provides manual sync API + smart detection to update initial_balance
when user deposits funds after trader creation.
Co-Authored-By: tinkle-community <tinklefund@gmail.com>
tinkle-community
0xYYBB | ZYY | Bobo
Professor-Chen
Rick
Diego
Lawrence Liu
Icy
WquGuru
Icyoung
0xYYBB | ZYY | Bobo
Shui
icy
hzb1115
Sue
PoorThoth
tangmengqiu