39b9e55434
perf: batch metadata query in KB retrieval to fix N+1 problem ( #5463 )
...
* perf: batch metadata query in KB retrieval to fix N+1 problem
Replace N sequential get_document_with_metadata() calls with a single
get_documents_with_metadata_batch() call using SQL IN clause.
Benchmark results (local SQLite):
- 10 docs: 10.67ms → 1.47ms (7.3x faster)
- 20 docs: 26.00ms → 2.68ms (9.7x faster)
- 50 docs: 63.87ms → 2.79ms (22.9x faster)
* refactor: use set[str] param type and chunk IN clause for SQLite safety
Address review feedback:
- Change doc_ids param from list[str] to set[str] to avoid unnecessary conversion
- Chunk IN clause into batches of 900 to stay under SQLite's 999 parameter limit
- Remove list() wrapping at call site, pass set directly
2026-02-26 09:59:37 +08:00
9c691b2266
chore: remove Electron desktop pipeline and switch to tauri repo ( #5226 )
...
* ci: remove Electron desktop build from release pipeline
* chore: remove electron desktop and switch to tauri release trigger
* ci: remove desktop workflow dispatch trigger
* refactor: migrate data paths to astrbot_path helpers
* fix: point desktop update prompt to AstrBot-desktop releases
2026-02-19 23:04:18 +09:00
7dd95d8a59
chore: auto ann fix by ruff ( #4903 )
...
* chore: auto fix by ruff
* refactor: 统一修正返回类型注解为 None/bool 以匹配实现
* refactor: 将 _get_next_page 改为异步并移除多余的请求错误抛出
* refactor: 将 get_client 的返回类型改为 object
* style: 为 LarkMessageEvent 的相关方法添加返回类型注解 None
---------
Co-authored-by: Soulter <37870767+Soulter@users.noreply.github.com >
2026-02-09 00:22:24 +08:00
9a91f2fb11
fix: ensure atomic creation of knowledge base with proper cleanup on failure ( #4406 )
...
* fix: ensure atomic creation of knowledge base with proper cleanup on failure
- Added pre-validation for embedding_provider_id parameter
- Added check for existing knowledge base with same name
- Implemented proper rollback mechanism when KBHelper initialization fails
- Uses same session for cleanup to ensure data consistency
- Fixes #4403
* fix: ensure atomic KB creation with session.flush() to remove race condition risks
* fix: ensure change the annotation back
2026-01-11 14:24:26 +08:00
792fb69d6d
perf: allow zero chunk overlap in recursive chunker ( #4258 )
...
* Allow zero chunk overlap
* Validate recursive chunking bounds
2025-12-30 15:23:05 +08:00
f624971613
chore: fix bunches of type checking errors ( #3213 )
...
* chore(core.utils): 🚨 修正错误Lint
* chore(core.provider): 🚨 修复基类错误Lint
* chore(core.utils): 补全session_get()的重载
* chore(core.provider): 🚨 修正实现错误Lint
* chore(core.platform): 🚨 修正platform基类和webchat的错误Lint
* chore(core.platform): 修正错误实现Lint
* fix(core.provider): 修复循环调用和错误assert
* chore(core.platform): 修复部分实现Lint
* chore(core.provider): 补充Dify.text_chat_stream的参数类型
* chore(core.pipeline): 🚨 修复错误Lint
* fix(core.slack): 补充遗漏导入
* chore(core.utils): 修复错误的session_get声明
* chore(core.platform): 移除Lark adapter import中的wildcard
* chore(core.db): 修复声明和部分逻辑
* chore(core.db): 添加typings，使faiss参数能被正确识别。
* chore(core): 修复声明
* chore(core): 修改声明
* chore: 补充faiss声明
* chore(dashboard): 修改实现，减少报错
* chore(package): 修改部分声明与实现，减少报错
* chore(core): 添加Handler的overload，以去除部分assert同时通过类型检查
* chore(core.pipeline): 修改Pipeline Scheduler的execute，将判断属性改为判断类型，通过静态类型检查
* chore(core.config): 添加类型标注，通过类型检查
* chore(core.message): 为File._download_file添加检查，通过类型检查
* fix: 将断言改为条件判断以实现优雅关闭的容错性
* refactor: 移除 discord 客户端中的 assert，改用 if None 判断并抛出异常
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: DiscordPlatformAdapter 对 self.client.user 为 None 做日志并返回，移除断言
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: 增强 Lark 相关空值/异常检查并完善日志输出
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* refactor: 将断言替换为条件检查并加入日志与错误处理
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* chore: 移除LLM生成的无用注释
* refactor: 使用 File.get_file 替换下载逻辑并移除 assert，提供默认 filename
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: Slack Socket 未初始化抛出运行时异常，图片 URL 判空改为非空判断
* refactor: 将 WeChatPadProAdapter 的断言改为空值判断并添加日志
* refactor: 使用 isinstance 替代断言实现类型判断，便于静态检查
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: 去除cast，直接使用字段与字典访问，修正端口解析
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* refactor: 使用 match-case 重构 ProviderManager 加载并通过类型检查抛出 TypeError
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: group_name_display 时若 group 对象为空则记录错误并返回
* fix: 将 _get_current_persona_id 的 assert 替换成 if guard 并返回 None
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: 优化插件目录存在性检查及图片URL非空验证，更新JSON排序配置
* fix: 将 datetime_str 的 assert 替换为显式检查并抛出异常
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* refactor: 移除 cast，改为运行时检查并在找不到调度器时跳过
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* refactor: 移除 cast，改用 isinstance 检查 FaissVecDB 并警告
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: 删除 typing.cast 导入，并在获取文件绝对路径前校验 file_
* refactor: 移除 typing.cast，简化内容安全检查调用
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* refactor: 将 PlatformMetadata.id 设为必填并在注册时传入 id，移除 cast
* refactor: 移除 cast，改用 HasInitialize 与 isinstance 进行初始化
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: 为 ProviderManager.initialize 增加ID类型判断，避免 None 导致 get 失败
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* refactor: 为 OTTSProvider 与 AzureNativeProvider 引入 _client 与 client 属性改进上下文管理
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: 为 Whisper 自托管源添加模型未初始化校验并直接调用 transcribe
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* refactor: 移除未使用的 cast 导入并简化 platform_name 赋值
* refactor: 引入 cast 并对 id 使用 cast(str, ...) 提升类型安全
* fix: 将 _id_to_sid 返回改为 str，空值返回空串；对 id 与 message_id 使用 cast
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* refactor: 重构 Discord 处理逻辑：强制 类型转换、优先斜杠指令并优化提及判断
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
* fix: 统一对 id 获取执行 cast，并在微信消息解析失败时抛错
* Revert "fix: 去除cast，直接使用字段与字典访问，修正端口解析"
This reverts commit 1cbfdf9d1baider@aider.chat >
* refactor: 调整 call_handler 泛型、Discord 通道注解及 FishAudioTTS API 请求类型
* refactor: 使用 col(...) 替代列引用并对结果进行 CursorResult 强转
* chore: ruff format
---------
Co-authored-by: aider (openai/gemini-3-pro-high) <aider@aider.chat >
Co-authored-by: Soulter <905617992@qq.com >
2025-12-09 14:13:47 +08:00
270c89c12f
feat: Add URL document parser for knowledge base ( #3622 )
...
* feat: 添加从 URL 上传文档的功能，支持进度回调和错误处理
* feat: 添加从 URL 上传文档的前端
* chore: 添加 URL 上传功能的警告提示，确保用户配置正确
* feat: 添加内容清洗功能，支持从 URL 上传文档时的清洗设置和服务提供商选择
* feat: 更新内容清洗系统提示，增强信息提取规则；添加 URL 上传功能的测试版标识
* style: format code
* perf: 优化上传设置，增强 URL 上传时的禁用逻辑和清洗提供商验证
* refactor:使用自带chunking模块
* refactor: 提取prompt到单独文件
* feat: 添加 Tavily API Key 配置对话框，增强网页搜索功能的配置体验
* fix: update URL hint and warning messages for clarity in knowledge base upload settings
* fix: 修复设置tavily_key的热重载问题
---------
Co-authored-by: Soulter <905617992@qq.com >
2025-11-17 19:05:14 +08:00
0b7fc29ac4
style: add ruff lint module of isort and pyupgrade, and some ruff check fix ( #3214 )
...
Co-authored-by: Dt8333 <25431943+Dt8333@users.noreply.github.com >
Co-authored-by: Soulter <905617992@qq.com >
2025-11-01 13:26:19 +08:00
0823f7aa48
在检查字面量集合的成员资格时使用 set
...
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
2025-10-25 22:04:17 +08:00
eb201c0420
feat: refactor knowledge base parsers and add MarkitdownParser for docx, xls, xlsx support
2025-10-25 22:00:54 +08:00
57f868cab1
Merge branch 'feature/knowledge-base' of https://github.com/lxfight/AstrBot into feature/knowledge-base
2025-10-25 13:53:03 +08:00
016783a1e5
feat: implement RecursiveCharacterChunker and update KnowledgeBaseManager to use it
2025-10-25 13:46:06 +08:00
594ccff9c8
fix: 添加数据库连接检查和知识库终止功能，增强错误处理和清理逻辑，修复知识库无法删除的问题
2025-10-25 11:56:37 +08:00
8f021eb35a
feat: refactor document storage to use SQLModel and enhance database operations
2025-10-24 23:17:37 +08:00
4cedc6d3c8
feat: add t-SNE visualization for FAISS index and enhance knowledge base retrieval with debug mode
2025-10-24 21:22:46 +08:00
4e9cce76da
feat: add timing logs for dense and sparse retrieval processes and adjust top K results in sparse retriever
2025-10-24 17:51:30 +08:00
9b004f3d2f
feat: update document retrieval to include limit and offset parameters
2025-10-24 17:38:22 +08:00
9430e3090d
feat: add progress callback for document upload and enhance upload progress tracking
2025-10-24 17:13:44 +08:00
ba44f9117b
feat: enhance document upload process with batch settings and improved chunk handling
2025-10-24 16:37:37 +08:00
38e3f27899
feat: update knowledge base retrieval configuration and UI adjustments
2025-10-24 15:06:07 +08:00
a6be0cc135
feat: refresh knowledge base and document after uploading a document
2025-10-24 14:28:27 +08:00
a53510bc41
refactor: comment out file path handling in KBHelper and search input in DocumentDetail
2025-10-24 14:27:01 +08:00
1fd482e899
feat: update chunk deletion to include document ID and refresh metadata
2025-10-24 14:18:32 +08:00
2f130ba009
feat: delete chunk and delete document
2025-10-24 13:59:17 +08:00
e0ac743cdb
perf: remove rerank functionality from settings tab and related form data
2025-10-24 12:13:51 +08:00
7e0a50fbf2
feat: enhance knowledge base retrieval with chunk metadata and pagination support; remove unused chunk model
2025-10-24 00:44:40 +08:00
59df244173
improve
2025-10-23 21:20:41 +08:00
e3aa1315ae
stage
2025-10-23 00:31:15 +08:00
65bc5efa19
feat: 集成知识库管理器，优化知识库上下文注入流程，移除冗余代码
2025-10-22 21:59:00 +08:00
a05868cc45
feat: 更新知识库管理器以支持重排序模型提供商，调整相关组件的默认配置和提示信息
2025-10-20 22:38:06 +08:00
2fc77aed15
feat: 添加知识库检索功能，支持根据知识库 ID 列出相关会话；更新相关界面和国际化文本
2025-10-20 22:23:35 +08:00
beccae933f
fix:修复KBSessionConfig的导入问题
2025-10-19 21:36:01 +08:00
a0254ed817
refactor: 优化知识库管理器和数据库操作的代码格式
2025-10-19 19:36:26 +08:00
ad96d676e6
feat: 实现知识库核心后端模块
...
- 实现完整的知识库数据模型（知识库、文档、文档块、会话配置）
- 实现基于 SQLite 的向量数据库存储和检索
- 实现文档解析器（PDF、TXT）和固定大小分块器
- 实现混合检索系统（密集向量检索 + BM25 稀疏检索 + RRF 融合）
- 实现知识库生命周期管理和消息注入器
- 支持会话级别的知识库配置和关联
2025-10-19 18:40:55 +08:00
CAICAII
エイカク
Dt8333
Li-shi-ling
Soulter
RC-CHN
LIghtJUNction
Soulter
lxfight