Commit Graph

82 Commits

Author SHA1 Message Date
catlog22
76ab4d67fe test(entities): add zero vector validation tests
Add comprehensive test coverage for zero and near-zero vector
detection in SemanticChunk embedding validation.

Solution-ID: SOL-20251228113612
Issue-ID: ISS-1766921318981-7
Task-ID: T2
2025-12-29 19:03:20 +08:00
catlog22
c859af1abf fix(entities): validate embeddings are non-zero vectors
Add L2 norm check to SemanticChunk.validate_embedding to reject
zero vectors. Prevents division by zero in cosine similarity
calculations downstream in vector search.

Solution-ID: SOL-20251228113612
Issue-ID: ISS-1766921318981-7
Task-ID: T1
2025-12-29 19:01:27 +08:00
catlog22
6a73d3c379 fix(search): handle path operation failures in symbol filtering
Adds robust exception handling for os.path.commonpath() in search_symbols()
to prevent crashes on malformed paths and Windows cross-drive scenarios.
Invalid symbols are skipped with debug logging, search continues.

Solution-ID: SOL-1735385400004
Issue-ID: ISS-1766921318981-4
Task-ID: T1
2025-12-29 18:59:10 +08:00
catlog22
5d5652c2c5 fix(sqlite-store): improve thread tracking in connection cleanup
Add fallback validation to detect dead threads missed by
threading.enumerate(), ensuring all stale connections are cleaned.

Solution-ID: SOL-1735392000002
Issue-ID: ISS-1766921318981-3
Task-ID: T2
2025-12-29 18:50:22 +08:00
catlog22
b958a1ea96 fix(sqlite-store): add periodic cleanup timer for connection pool
Implement background timer to proactively clean stale connections
every 5 minutes, preventing indefinite accumulation.

Solution-ID: SOL-1735392000002
Issue-ID: ISS-1766921318981-3
Task-ID: T1
2025-12-29 18:43:55 +08:00
catlog22
9a45732a39 test(codex-lens): add connection pool stress tests
Solution-ID: SOL-1735410004
Issue-ID: ISS-1766921318981-24
Task-ID: T3
2025-12-29 18:16:03 +08:00
catlog22
015b46e58b test(codex-lens): add concurrent write operation tests
Solution-ID: SOL-1735410004
Issue-ID: ISS-1766921318981-24
Task-ID: T2
2025-12-29 18:12:09 +08:00
catlog22
042a99dbe3 test(codex-lens): add concurrent read operation tests
Solution-ID: SOL-1735410004

Issue-ID: ISS-1766921318981-24

Task-ID: T1
2025-12-29 17:59:08 +08:00
catlog22
1396010437 fix(embedder): add lock protection for cache read operations
Protect fast path cache read in get_embedder() to prevent KeyError

during concurrent access and cache clearing operations.

Solution-ID: SOL-1735392000001

Issue-ID: ISS-1766921318981-2

Task-ID: T1
2025-12-29 12:33:23 +08:00
catlog22
84d06f4273 fix(registry): normalize path case for comparison on Windows
Adds case normalization for path comparison on Windows to handle
case-insensitive filesystem behavior. Preserves case-sensitivity on Unix.

Fixes: ISS-1766921318981-13

Solution-ID: SOL-1735386000-13
Issue-ID: ISS-1766921318981-13
Task-ID: T1
2025-12-28 21:51:23 +08:00
catlog22
18cc536f65 refactor(vector-store): use consistent EPSILON constant
Define module-level EPSILON constant and use it in both
_cosine_similarity and _refresh_cache for consistent
floating point precision handling.

Solution-ID: SOL-20251228113619
Issue-ID: ISS-1766921318981-11
Task-ID: T3
2025-12-28 21:40:46 +08:00
catlog22
af2ff54cb7 test(vector-store): add epsilon tolerance edge case tests
Add comprehensive test coverage for near-zero norms, product
underflow, and floating point precision edge cases in
_cosine_similarity function.

Solution-ID: SOL-20251228113619
Issue-ID: ISS-1766921318981-11
Task-ID: T2
2025-12-28 21:37:59 +08:00
catlog22
6486c56850 fix(vector-store): add epsilon tolerance for norm checks
Replace exact zero comparison with epsilon-based check (< 1e-10)
in _cosine_similarity to handle floating point precision issues.
Also check for product underflow to prevent inf/nan from division
by very small numbers.

Solution-ID: SOL-20251228113619
Issue-ID: ISS-1766921318981-11
Task-ID: T1
2025-12-28 21:11:30 +08:00
catlog22
93dcdd2293 fix(config): log configuration loading errors instead of silently ignoring
Replaces bare exception handler in load_settings() with logging.warning()
to help users debug configuration file issues (syntax errors, permissions).
Maintains backward compatibility - errors do not break initialization.

Solution-ID: SOL-1735385400001
Issue-ID: ISS-1766921318981-1
Task-ID: T1
2025-12-28 21:06:23 +08:00
catlog22
58caccb250 test(ranking): add edge case tests for normalize_weights
Add comprehensive test coverage for NaN, infinity, and all-None
edge cases in weight normalization to prevent regression.

Solution-ID: SOL-20251228113631
Issue-ID: ISS-1766921318981-0
Task-ID: T2
2025-12-28 20:59:08 +08:00
catlog22
598eed92cb fix(ranking): add explicit NaN check in normalize_weights
Add math.isnan() check before math.isfinite() to properly catch
NaN values in weight totals. Prevents division by NaN which could
produce unexpected results in RRF fusion calculations.

Solution-ID: SOL-20251228113631
Issue-ID: ISS-1766921318981-0
Task-ID: T1
2025-12-28 20:55:03 +08:00
catlog22
a2c88ba885 feat: Add project guidelines support and enhance project overview rendering 2025-12-28 14:50:50 +08:00
catlog22
4061ae48c4 feat: Implement adaptive RRF weights and query intent detection
- Added integration tests for adaptive RRF weights in hybrid search.
- Enhanced query intent detection with new classifications: keyword, semantic, and mixed.
- Introduced symbol boosting in search results based on explicit symbol matches.
- Implemented embedding-based reranking with configurable options.
- Added global symbol index for efficient symbol lookups across projects.
- Improved file deletion handling on Windows to avoid permission errors.
- Updated chunk configuration to increase overlap for better context.
- Modified package.json test script to target specific test files.
- Created comprehensive writing style guidelines for documentation.
- Added TypeScript tests for query intent detection and adaptive weights.
- Established performance benchmarks for global symbol indexing.
2025-12-26 15:08:47 +08:00
catlog22
3b842ed290 feat(cli-executor): add streaming option and enhance output handling
- Introduced a `stream` parameter to control output streaming vs. caching.
- Enhanced status determination logic to prioritize valid output over exit codes.
- Updated output structure to include full stdout and stderr when not streaming.

feat(cli-history-store): extend conversation turn schema and migration

- Added `cached`, `stdout_full`, and `stderr_full` fields to the conversation turn schema.
- Implemented database migration to add new columns if they do not exist.
- Updated upsert logic to handle new fields.

feat(codex-lens): implement global symbol index for fast lookups

- Created `GlobalSymbolIndex` class to manage project-wide symbol indexing.
- Added methods for adding, updating, and deleting symbols in the global index.
- Integrated global index updates into directory indexing processes.

feat(codex-lens): optimize search functionality with global index

- Enhanced `ChainSearchEngine` to utilize the global symbol index for faster searches.
- Added configuration option to enable/disable global symbol indexing.
- Updated tests to validate global index functionality and performance.
2025-12-25 22:22:31 +08:00
catlog22
203100431b feat: 添加 Code Index MCP 提供者支持,更新相关 API 和配置 2025-12-25 19:58:42 +08:00
catlog22
ebcbb11cb2 feat: Enhance CodexLens search functionality with new parameters and result handling
- Added search limit, content length, and extra files input fields in the CodexLens manager UI.
- Updated API request parameters to include new fields: max_content_length and extra_files_count.
- Refactored smart-search.ts to support new parameters with default values.
- Implemented result splitting logic to return both full content and additional file paths.
- Updated CLI commands to remove worker limits and allow dynamic scaling based on endpoint count.
- Introduced EmbeddingPoolConfig for improved embedding management and auto-discovery of providers.
- Enhanced search engines to utilize new parameters for fuzzy and exact searches.
- Added support for embedding single texts in the LiteLLM embedder.
2025-12-25 16:16:44 +08:00
catlog22
a1413dd1b3 feat: Unified Embedding Pool with auto-discovery
Architecture refactoring for multi-provider rotation:

Backend:
- Add EmbeddingPoolConfig type with autoDiscover support
- Implement discoverProvidersForModel() for auto-aggregation
- Add GET/PUT /api/litellm-api/embedding-pool endpoints
- Add GET /api/litellm-api/embedding-pool/discover/:model preview
- Convert ccw-litellm status check to async with 5-min cache
- Maintain backward compatibility with legacy rotation config

Frontend:
- Add "Embedding Pool" tab in API Settings
- Auto-discover providers when target model selected
- Show provider/key count with include/exclude controls
- Increase sidebar width (280px → 320px)
- Add sync result feedback on save

Other:
- Remove worker count limits (was max=32)
- Add i18n translations (EN/CN)
- Update .gitignore for .mcp.json

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-25 16:06:49 +08:00
catlog22
8e744597d1 feat: Implement CodexLens multi-provider embedding rotation management
- Added functions to get and update CodexLens embedding rotation configuration.
- Introduced functionality to retrieve enabled embedding providers for rotation.
- Created endpoints for managing rotation configuration via API.
- Enhanced dashboard UI to support multi-provider rotation configuration.
- Updated internationalization strings for new rotation features.
- Adjusted CLI commands and embedding manager to support increased concurrency limits.
- Modified hybrid search weights for improved ranking behavior.
2025-12-25 14:13:27 +08:00
catlog22
501d9a05d4 fix: 修复 ModelScope API 路由 bug 导致的 Ollama 连接错误
- 添加 _sanitize_text() 方法处理以 'import' 开头的文本
- ModelScope 后端错误地将此类文本路由到本地 Ollama 端点
- 通过在文本前添加空格绕过路由检测,不影响嵌入质量
- 增强 embedding_manager.py 的重试逻辑和错误处理
- 在 commands.py 中成功生成后调用全局模型锁定

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-25 12:52:43 +08:00
catlog22
229d51cd18 feat: 添加全局模型锁定功能,防止不同模型混合使用,增强嵌入生成的稳定性 2025-12-25 11:20:05 +08:00
catlog22
40e61b30d6 feat: 添加多端点支持和负载均衡功能,增强 LiteLLM 嵌入管理 2025-12-25 11:01:08 +08:00
catlog22
3c3ce55842 feat: 添加对 LiteLLM 嵌入后端的支持,增强并发 API 调用能力 2025-12-24 22:20:13 +08:00
catlog22
e3e61bcae9 feat: Enhance LiteLLM integration and CLI management
- Added token estimation and batching functionality in LiteLLMEmbedder to handle large text inputs efficiently.
- Updated embed method to support max_tokens_per_batch parameter for better API call management.
- Introduced new API routes for managing custom CLI endpoints, including GET, POST, PUT, and DELETE methods.
- Enhanced CLI history component to support source directory context for native session content.
- Improved error handling and logging in various components for better debugging and user feedback.
- Added internationalization support for new API endpoint features in the i18n module.
- Updated CodexLens CLI commands to allow for concurrent API calls with a max_workers option.
- Enhanced embedding manager to track model information and handle embeddings generation more robustly.
- Added entry points for CLI commands in the package configuration.
2025-12-24 18:01:26 +08:00
catlog22
e671b45948 feat: Enhance configuration management and embedding capabilities
- Added JSON-based settings management in Config class for embedding and LLM configurations.
- Introduced methods to save and load settings from a JSON file.
- Updated BaseEmbedder and its subclasses to include max_tokens property for better token management.
- Enhanced chunking strategy to support recursive splitting of large symbols with improved overlap handling.
- Implemented comprehensive tests for recursive splitting and chunking behavior.
- Added CLI tools configuration management for better integration with external tools.
- Introduced a new command for compacting session memory into structured text for recovery.
2025-12-24 16:32:27 +08:00
catlog22
b00113d212 feat: Enhance embedding management and model configuration
- Updated embedding_manager.py to include backend parameter in model configuration.
- Modified model_manager.py to utilize cache_name for ONNX models.
- Refactored hybrid_search.py to improve embedder initialization based on backend type.
- Added backend column to vector_store.py for better model configuration management.
- Implemented migration for existing database to include backend information.
- Enhanced API settings implementation with comprehensive provider and endpoint management.
- Introduced LiteLLM integration guide detailing configuration and usage.
- Added examples for LiteLLM usage in TypeScript.
2025-12-24 14:03:59 +08:00
catlog22
46ac591fe8 Merge branch 'main' of https://github.com/catlog22/Claude-Code-Workflow 2025-12-23 20:46:01 +08:00
catlog22
bf66b095c7 feat: Add unified LiteLLM API management with dashboard UI and CLI integration
- Create ccw-litellm Python package with AbstractEmbedder and AbstractLLMClient interfaces
- Add BaseEmbedder abstraction and factory pattern to codex-lens for pluggable backends
- Implement API Settings dashboard page for provider credentials and custom endpoints
- Add REST API routes for CRUD operations on providers and endpoints
- Extend CLI with --model parameter for custom endpoint routing
- Integrate existing context-cache for @pattern file resolution
- Add provider model registry with predefined models per provider type
- Include i18n translations (en/zh) for all new UI elements

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 20:36:32 +08:00
catlog22
39056292b7 feat: Add CodexLens Manager to dashboard and enhance GPU management
- Introduced a new CodexLens Manager item in the dashboard for easier access.
- Implemented GPU management commands in the CLI, including listing available GPUs, selecting a specific GPU, and resetting to automatic detection.
- Enhanced the embedding generation process to utilize GPU resources more effectively, including batch size optimization for better performance.
- Updated the embedder to support device ID options for GPU selection, ensuring compatibility with DirectML and CUDA.
- Added detailed logging and error handling for GPU detection and selection processes.
- Updated package version to 6.2.9 and added comprehensive documentation for Codex Agent Execution Protocol.
2025-12-23 18:35:30 +08:00
rhyme
1998f3ae8a fix(codexlens): correct fastembed 0.7.4 cache path and download trigger
- Update cache path to ~/.cache/huggingface (HuggingFace Hub default)
- Fix model path format: models--{org}--{model}
- Add .embed() call to trigger actual download in download_model()
- Ensure cross-platform compatibility (Linux/Windows)
2025-12-23 14:51:08 +08:00
catlog22
3cd842ca1a fix: ccw package.json removal - add root build script and fix cli.ts path resolution
- Fix cli.ts loadPackageInfo() to try root package.json first (../../package.json)
- Add build script and devDependencies to root package.json
- Remove ccw/package.json and ccw/package-lock.json (no longer needed)
- CodexLens: add config.json support for index_dir configuration

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 10:25:15 +08:00
catlog22
8203d690cb fix: CodexLens model detection, hybrid search stability, and JSON logging
- Fix model installation detection using fastembed ONNX cache names
- Add embeddings_config table for model metadata tracking
- Fix hybrid search segfault by using single-threaded GPU mode
- Suppress INFO logs in JSON mode to prevent error display
- Add model dropdown filtering to show only installed models

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-22 21:49:10 +08:00
catlog22
cf58dc0dd3 bump version to 6.2.6 in package.json 2025-12-22 20:17:38 +08:00
catlog22
6a69af3bf1 feat: 更新嵌入批处理大小至 256,以优化性能并提高 GPU 加速效率 2025-12-22 17:55:05 +08:00
catlog22
acdfbb4644 feat: Enhance CodexLens with GPU support and semantic status improvements
- Added accelerator and providers fields to SemanticStatus interface.
- Updated checkSemanticStatus function to retrieve ONNX providers and accelerator type.
- Introduced detectGpuSupport function to identify available GPU modes (CUDA, DirectML).
- Modified installSemantic function to support GPU acceleration modes and clean up ONNX Runtime installations.
- Updated package requirements in PKG-INFO for semantic-gpu and semantic-directml extras.
- Added new source files for GPU support and enrichment functionalities.
- Updated tests to cover new features and ensure comprehensive testing.
2025-12-22 17:42:26 +08:00
catlog22
72f24bf535 feat: 更新版本号至 6.2.4,添加 GPU 加速支持和相关依赖 2025-12-22 14:15:36 +08:00
catlog22
e60d793c8c fix: 修复 SmartSearch 的 ripgrep limit 和 FTS 分词器问题
- Ripgrep 模式: 添加总结果数量限制,防止返回超过 2MB 数据
  - --max-count 只限制每个文件的匹配数,现在在收集结果时应用 limit
  - 达到限制时在 metadata 中添加 warning 提示

- FTS 分词器: 将点号(.)添加到 tokenchars,修复 PortRole.FLOW 等带点号标识符的精确搜索
  - 更新 dir_index.py 和 migration_004_dual_fts.py 中的 tokenize 配置
  - 需要重建索引才能生效

- Exact 模式: 添加 fuzzy 回退,当精确搜索无结果时自动尝试模糊搜索
  - 回退时在 metadata 中标注 fallback: 'fuzzy'

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-22 09:50:29 +08:00
catlog22
fc4a9af0cb feat: 引入流式生成器以优化内存使用,改进嵌入生成过程 2025-12-21 23:47:29 +08:00
catlog22
fa64e11a77 refactor: 优化嵌入生成过程,调整批处理大小和内存管理策略 2025-12-21 23:37:34 +08:00
catlog22
210f0f1012 feat: 添加钩子命令,简化 Claude Code 钩子操作接口,支持会话上下文加载和通知功能 2025-12-21 23:28:19 +08:00
catlog22
2871950ab8 fix: 修复向量索引进度显示过早完成的问题
问题:FTS 索引完成后立即显示 100%,但嵌入生成仍在后台运行

修复:
- codex-lens.ts: 将 "Indexed X files" 阶段从 complete 改为 fts_complete (60%)
- codex-lens.ts: 添加嵌入批次和 Finalizing index 阶段解析
- embedding_manager.py: 使用 bulk_insert() 模式延迟 ANN 索引构建
- embedding_manager.py: 添加 "Finalizing index" 进度回调

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 20:55:45 +08:00
catlog22
5849f751bc fix: 修复嵌入生成内存泄漏,优化性能
- HNSW 索引:预分配从 100 万降至 5 万,添加动态扩容和可控保存
- Embedder:添加 embed_to_numpy() 避免 .tolist() 转换,增强缓存清理
- embedding_manager:每 10 批次重建 embedder 实例,显式 gc.collect()
- VectorStore:添加 bulk_insert() 上下文管理器,支持 numpy 批量写入
- Chunker:添加 skip_token_count 轻量模式,使用 char/4 估算(~9x 加速)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 19:15:47 +08:00
catlog22
f492f4839a refactor: 移除 CLI 中过宽的异常捕获
- 移除所有 16 个 except Exception 块
- 只保留对特定异常的捕获 (StorageError, ConfigError, SearchError 等)
- 允许未知异常自然传播,便于调试
- 保留嵌入功能的可选异常处理

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 17:19:54 +08:00
catlog22
fa81793bea refactor: 优化异常处理,使用 cached_property 替代 property,增强代码可读性;添加 RelationshipType 枚举以规范化关系类型 2025-12-21 17:01:49 +08:00
catlog22
6eebdb8898 fix: 修复额外的内存泄露问题
1. hybrid_search.py: 修复 _search_vector 方法中的 SQLite 连接泄露
   - 使用 with 语句包装数据库连接
   - 添加异常处理确保连接正确关闭

2. symbol_extractor.py: 添加上下文管理器支持
   - 实现 __enter__ 和 __exit__ 方法
   - 支持 with 语句自动管理资源

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 16:39:38 +08:00
catlog22
3e9a309079 refactor: 移除图索引功能,修复内存泄露,优化嵌入生成
主要更改:

1. 移除图索引功能 (graph indexing)
   - 删除 graph_analyzer.py 及相关迁移文件
   - 移除 CLI 的 graph 命令和 --enrich 标志
   - 清理 chain_search.py 中的图查询方法 (370行)
   - 删除相关测试文件

2. 修复嵌入生成内存问题
   - 重构 generate_embeddings.py 使用流式批处理
   - 改用 embedding_manager 的内存安全实现
   - 文件从 548 行精简到 259 行 (52.7% 减少)

3. 修复内存泄露
   - chain_search.py: quick_search 使用 with 语句管理 ChainSearchEngine
   - embedding_manager.py: 使用 with 语句管理 VectorStore
   - vector_store.py: 添加暴力搜索内存警告

4. 代码清理
   - 移除 Symbol 模型的 token_count 和 symbol_type 字段
   - 清理相关测试用例

测试: 760 passed, 7 skipped

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 16:22:03 +08:00