Claude-Code-Workflow/codex-lens/docs/REAL_LSP_SERVER_PLAN.md

CodexLens Real LSP Server Implementation Plan

Version: 2.0
Status: Ready for Implementation
Based on: Existing LSP_INTEGRATION_PLAN.md + Real Language Server Integration
Goal: Implement true LSP server functionality (like cclsp), not pre-indexed search

---

Executive Summary

Current State vs Target State

Aspect	Current (Pre-indexed)	Target (Real LSP)
Data Source	Cached database index	Live language servers
Freshness	Stale (depends on re-index)	Real-time (LSP protocol)
Accuracy	Good for indexed content	Perfect (from language server)
Latency	<50ms (database)	~50-200ms (LSP)
Language Support	Limited to parsed symbols	Full LSP support (all languages)
Complexity	Simple (DB queries)	High (LSP protocol + server mgmt)

Why Real LSP vs Index-Based

Problem with current approach:

• 符号搜索与smart_search没有本质区别
• 依赖预索引数据，不能实时反映代码变化
• 不支持advanced LSP功能(rename, code actions等)

Advantages of real LSP:

• ✅ Real-time code intelligence
• ✅ Supported by all major IDEs (VSCode, Neovim, Sublime, etc.)
• ✅ Standard protocol (Language Server Protocol)
• ✅ Advanced features: rename, code actions, formatting
• ✅ Language-agnostic (TypeScript, Python, Go, Rust, Java, etc.)

---

Architecture Design

System Architecture

┌─────────────────────────────────────────────────────────┐
│                    Client Layer                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │   VS Code    │  │    Neovim    │  │   Sublime    │  │
│  │ (LSP Client) │  │ (LSP Client) │  │ (LSP Client) │  │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  │
│         │                 │                 │           │
└─────────┼─────────────────┼─────────────────┼───────────┘
          │ LSP Protocol    │                 │
          │ (JSON-RPC/stdio)│                 │
┌─────────▼─────────────────▼─────────────────▼───────────┐
│            CodexLens LSP Server Bridge                   │
│  ┌─────────────────────────────────────────────────────┐ │
│  │  LSP Protocol Handler (pygls)                       │ │
│  │  • initialize / shutdown                            │ │
│  │  • textDocument/definition                          │ │
│  │  • textDocument/references                          │ │
│  │  • textDocument/hover                               │ │
│  │  • textDocument/completion                          │ │
│  │  • textDocument/formatting                          │ │
│  │  • workspace/symbol                                 │ │
│  └────────────────────┬────────────────────────────────┘ │
│                       │                                   │
│  ┌────────────────────▼────────────────────────────────┐ │
│  │  Language Server Multiplexer                        │ │
│  │  • File type routing (ts→tsserver, py→pylsp, etc.)  │ │
│  │  • Multi-server management                          │ │
│  │  • Request forwarding & response formatting         │ │
│  └────────────────────┬────────────────────────────────┘ │
│                       │                                   │
│  ┌────────────────────▼────────────────────────────────┐ │
│  │  Language Servers (Spawned)                         │ │
│  │  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐   │ │
│  │  │tsserver│  │ pylsp  │  │ gopls  │  │rust-   │   │ │
│  │  │        │  │        │  │        │  │analyzer│   │ │
│  │  └────────┘  └────────┘  └────────┘  └────────┘   │ │
│  └─────────────────────────────────────────────────────┘ │
│                                                           │
│  ┌─────────────────────────────────────────────────────┐ │
│  │  Codex-Lens Core (Optional - MCP Layer)             │ │
│  │  • Semantic search                                  │ │
│  │  • Custom MCP tools (enrich_prompt, etc.)           │ │
│  │  • Hook system (pre-tool, post-tool)                │ │
│  └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘

Key Differences from Index-Based Approach

1. Request Flow

   • Index: Query → Database → Results
   • LSP: Request → Route to LS → LS processes live code → Results

2. Configuration

   • Index: Depends on indexing state
   • LSP: Depends on installed language servers

3. Latency Profile

   • Index: Consistent (~50ms)
   • LSP: Variable (50-500ms depending on LS performance)

---

Implementation Phases

Phase 1: LSP Server Bridge (Foundation)

Duration: ~3-5 days
Complexity: Medium
Dependencies: pygls library

1.1 Setup & Dependencies

File: pyproject.toml

[project.optional-dependencies]
lsp = [
    "pygls>=1.3.0",
    "lsprotocol>=2023.0.0",
]

[project.scripts]
codexlens-lsp = "codexlens.lsp.server:main"

Installation:

pip install -e ".[lsp]"

1.2 LSP Server Core

Files to create:

1. src/codexlens/lsp/__init__.py - Package init
2. src/codexlens/lsp/server.py - Server entry point
3. src/codexlens/lsp/multiplexer.py - LS routing & management
4. src/codexlens/lsp/handlers.py - LSP request handlers

Key responsibilities:

• Initialize LSP server via pygls
• Handle client capabilities negotiation
• Route requests to appropriate language servers
• Format language server responses to LSP format

1.3 Acceptance Criteria

• Server starts with codexlens-lsp --stdio
• Responds to initialize request
• Spawns language servers on demand
• Handles shutdown cleanly
• No crashes on malformed requests

---

Phase 2: Language Server Multiplexer

Duration: ~5-7 days
Complexity: High
Dependencies: Phase 1 complete

2.1 Multi-Server Management

File: src/codexlens/lsp/multiplexer.py

Responsibilities:

• Spawn language servers based on file extension
• Maintain server process lifecycle
• Route requests by document type
• Handle server crashes & restarts

Supported Language Servers:

Language	Server	Installation
TypeScript	typescript-language-server	npm i -g typescript-language-server
Python	pylsp	pip install python-lsp-server
Go	gopls	go install golang.org/x/tools/gopls@latest
Rust	rust-analyzer	rustup component add rust-analyzer
Java	jdtls	Download JDTLS
C/C++	clangd	apt install clangd

2.2 Configuration

File: codexlens-lsp.json (user config)

{
  "languageServers": {
    "typescript": {
      "command": ["typescript-language-server", "--stdio"],
      "extensions": ["ts", "tsx", "js", "jsx"],
      "rootDir": "."
    },
    "python": {
      "command": ["pylsp"],
      "extensions": ["py", "pyi"],
      "rootDir": ".",
      "settings": {
        "pylsp": {
          "plugins": {
            "pycodestyle": { "enabled": true },
            "pylint": { "enabled": false }
          }
        }
      }
    },
    "go": {
      "command": ["gopls"],
      "extensions": ["go"],
      "rootDir": "."
    },
    "rust": {
      "command": ["rust-analyzer"],
      "extensions": ["rs"],
      "rootDir": "."
    }
  },
  "debug": false,
  "logLevel": "info"
}

2.3 Acceptance Criteria

• Routes requests to correct LS based on file type
• Spawns servers on first request
• Reuses existing server instances
• Handles server restarts on crash
• Respects initialization options from config

---

Phase 3: Core LSP Handlers

Duration: ~5-7 days
Complexity: Medium
Dependencies: Phase 1-2 complete

3.1 Essential Handlers

Implement LSP request handlers for core functionality:

Handler Mapping:

Handlers = {
    # Navigation
    "textDocument/definition": handle_definition,
    "textDocument/references": handle_references,
    "textDocument/declaration": handle_declaration,
    
    # Hover & Info
    "textDocument/hover": handle_hover,
    "textDocument/signatureHelp": handle_signature_help,
    
    # Completion
    "textDocument/completion": handle_completion,
    "completionItem/resolve": handle_completion_resolve,
    
    # Symbols
    "textDocument/documentSymbol": handle_document_symbols,
    "workspace/symbol": handle_workspace_symbols,
    
    # Editing
    "textDocument/formatting": handle_formatting,
    "textDocument/rangeFormatting": handle_range_formatting,
    "textDocument/rename": handle_rename,
    
    # Diagnostics
    "textDocument/publishDiagnostics": handle_publish_diagnostics,
    
    # Misc
    "textDocument/codeAction": handle_code_action,
    "textDocument/codeLens": handle_code_lens,
}

3.2 Request Forwarding Logic

def forward_request_to_lsp(handler_name, params):
    """Forward request to appropriate language server."""
    
    # Extract document info
    document_uri = params.get("textDocument", {}).get("uri")
    file_ext = extract_extension(document_uri)
    
    # Get language server
    ls = multiplexer.get_server(file_ext)
    if not ls:
        return {"error": f"No LS for {file_ext}"}
    
    # Convert position (1-based → 0-based)
    normalized_params = normalize_positions(params)
    
    # Forward to LS
    response = ls.send_request(handler_name, normalized_params)
    
    # Convert response format
    return normalize_response(response)

3.3 Acceptance Criteria

• All handlers implemented and tested
• Proper position coordinate conversion (LSP is 0-based, user-facing is 1-based)
• Error handling for missing language servers
• Response formatting matches LSP spec
• Latency < 500ms for 95th percentile

---

Phase 4: Advanced Features

Duration: ~3-5 days
Complexity: Medium
Dependencies: Phase 1-3 complete

4.1 Position Tolerance (cclsp-like feature)

Some LSP clients (like Claude Code with fuzzy positions) may send imprecise positions. Implement retry logic:

def find_symbol_with_tolerance(ls, uri, position, max_attempts=5):
    """Try multiple position offsets if exact position fails."""
    
    positions_to_try = [
        position,                              # Original
        (position.line - 1, position.char),   # One line up
        (position.line + 1, position.char),   # One line down
        (position.line, max(0, position.char - 1)),  # One char left
        (position.line, position.char + 1),   # One char right
    ]
    
    for pos in positions_to_try:
        try:
            result = ls.send_request("textDocument/definition", {
                "textDocument": {"uri": uri},
                "position": pos
            })
            if result:
                return result
        except:
            continue
    
    return None

4.2 MCP Integration (Optional)

Extend with MCP provider for Claude Code hooks:

class MCPBridgeHandler:
    """Bridge LSP results into MCP context."""
    
    def build_mcp_context_from_lsp(self, symbol_name, lsp_results):
        """Convert LSP responses to MCP context."""
        # Implementation
        pass

4.3 Acceptance Criteria

• Position tolerance working (≥3 positions tried)
• MCP context generation functional
• Hook system integration complete
• All test coverage > 80%

---

Phase 5: Deployment & Documentation

Duration: ~2-3 days
Complexity: Low
Dependencies: Phase 1-4 complete

5.1 Installation & Setup Guide

Create comprehensive documentation:

• Installation instructions for each supported language
• Configuration guide
• Troubleshooting
• Performance tuning

5.2 CLI Tools

# Start LSP server
codexlens-lsp --stdio

# Check configured language servers
codexlens-lsp --list-servers

# Validate configuration
codexlens-lsp --validate-config

# Show logs
codexlens-lsp --log-level debug

5.3 Acceptance Criteria

• Documentation complete with examples
• All CLI commands working
• Integration tested with VS Code, Neovim
• Performance benchmarks documented

---

Module Structure

src/codexlens/lsp/
├── __init__.py                 # Package exports
├── server.py                   # LSP server entry point
├── multiplexer.py              # Language server manager
├── handlers.py                 # LSP request handlers
├── position_utils.py           # Coordinate conversion utilities
├── process_manager.py          # Language server process lifecycle
├── response_formatter.py        # LSP response formatting
└── config.py                   # Configuration loading

tests/lsp/
├── test_multiplexer.py         # LS routing tests
├── test_handlers.py            # Handler tests
├── test_position_conversion.py  # Coordinate tests
├── test_integration.py         # Full LSP handshake
└── fixtures/
    ├── sample_python.py        # Test files
    └── sample_typescript.ts

---

Dependency Graph

Phase 5 (Deployment)
    ↑
Phase 4 (Advanced Features)
    ↑
Phase 3 (Core Handlers)
    ├─ Depends on: Phase 2
    ├─ Depends on: Phase 1
    └─ Deliverable: Full LSP functionality
    
Phase 2 (Multiplexer)
    ├─ Depends on: Phase 1
    └─ Deliverable: Multi-server routing
    
Phase 1 (Server Bridge)
    └─ Deliverable: Basic LSP server

---

Technology Stack

Component	Technology	Rationale
LSP Implementation	pygls	Mature, well-maintained
Protocol	LSP 3.17+	Latest stable version
Process Management	subprocess + psutil	Standard Python, no external deps
Configuration	JSON	Simple, widely understood
Logging	logging module	Built-in, standard
Testing	pytest + pytest-asyncio	Industry standard

---

Risk Assessment

Risk	Probability	Impact	Mitigation
Language server crashes	Medium	High	Auto-restart with exponential backoff
Configuration errors	Medium	Medium	Validation on startup
Performance degradation	Low	High	Implement caching + benchmarks
Position mismatch issues	Medium	Low	Tolerance layer (try multiple positions)
Memory leaks (long sessions)	Low	Medium	Connection pooling + cleanup timers

---

Success Metrics

1. Functionality: All 7 core LSP handlers working
2. Performance: p95 latency < 500ms for typical requests
3. Reliability: 99.9% uptime in production
4. Coverage: >80% code coverage
5. Documentation: Complete with examples
6. Multi-language: Support for 5+ languages

---

Comparison: This Approach vs Alternatives

Option A: Real LSP Server (This Plan) ✅ RECOMMENDED

Pros:

• ✅ True real-time code intelligence
• ✅ Supports all LSP clients (VSCode, Neovim, Sublime, Emacs, etc.)
• ✅ Advanced features (rename, code actions, formatting)
• ✅ Language-agnostic
• ✅ Follows industry standard protocol

Cons:

• ❌ More complex implementation
• ❌ Depends on external language servers
• ❌ Higher latency than index-based

Effort: ~20-25 days

---

Option B: Enhanced Index-Based (Current Approach)

Pros:

• ✅ Simple implementation
• ✅ Fast (<50ms)
• ✅ No external dependencies

Cons:

• ❌ Same as smart_search (user's concern)
• ❌ Stale data between re-indexes
• ❌ Limited to indexed symbols
• ❌ No advanced LSP features

Effort: ~5-10 days

---

Option C: Hybrid (LSP + Index)

Pros:

• ✅ Real-time from LSP
• ✅ Fallback to index
• ✅ Best of both worlds

Cons:

• ❌ Highest complexity
• ❌ Difficult to debug conflicts
• ❌ Higher maintenance burden

Effort: ~30-35 days

---

Next Steps

1. Approve Plan: Confirm this approach matches requirements
2. Setup Dev Environment: Install language servers
3. Phase 1 Implementation: Start with server bridge
4. Iterative Testing: Test each phase with real IDE integration
5. Documentation: Maintain docs as implementation progresses

---

---

Appendix A: VSCode Bridge Implementation

A.1 Overview

VSCode Bridge 是另一种集成方式，通过VSCode扩展暴露其内置LSP功能给外部工具（如CCW MCP Server）。

Architecture:

┌─────────────────────────────────────────────────────────────────┐
│                        Claude Code / CCW                         │
│                     (MCP Client / CLI)                           │
└───────────────────────────┬─────────────────────────────────────┘
                            │
                            │ MCP Tool Call (vscode_lsp)
                            │
┌───────────────────────────▼─────────────────────────────────────┐
│                      CCW MCP Server                              │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │                    vscode_lsp Tool                          │ │
│  │   • HTTP client to VSCode Bridge                            │ │
│  │   • Parameter validation (Zod)                              │ │
│  │   • Response formatting                                      │ │
│  └────────────────────────┬────────────────────────────────────┘ │
└───────────────────────────┼─────────────────────────────────────┘
                            │
                            │ HTTP POST (localhost:3457)
                            │
┌───────────────────────────▼─────────────────────────────────────┐
│                   ccw-vscode-bridge Extension                    │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │                   HTTP Server (port 3457)                   │ │
│  │   Endpoints:                                                 │ │
│  │   • POST /get_definition                                     │ │
│  │   • POST /get_references                                     │ │
│  │   • POST /get_hover                                          │ │
│  │   • POST /get_document_symbols                               │ │
│  └────────────────────────┬────────────────────────────────────┘ │
│                           │                                      │
│  ┌────────────────────────▼────────────────────────────────────┐ │
│  │                  VSCode API Calls                            │ │
│  │   vscode.commands.executeCommand():                         │ │
│  │   • vscode.executeDefinitionProvider                        │ │
│  │   • vscode.executeReferenceProvider                         │ │
│  │   • vscode.executeHoverProvider                             │ │
│  │   • vscode.executeDocumentSymbolProvider                    │ │
│  └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
                            │
                            │ VSCode LSP Integration
                            │
┌───────────────────────────▼─────────────────────────────────────┐
│                 VSCode Language Services                         │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐            │
│  │TypeScript│  │ Python  │  │   Go    │  │  Rust   │            │
│  │  Server  │  │  Server │  │ (gopls) │  │Analyzer │            │
│  └─────────┘  └─────────┘  └─────────┘  └─────────┘            │
└─────────────────────────────────────────────────────────────────┘

A.2 Component Files

已创建的文件:

1. ccw-vscode-bridge/package.json - VSCode扩展配置
2. ccw-vscode-bridge/tsconfig.json - TypeScript配置
3. ccw-vscode-bridge/src/extension.ts - 扩展主代码
4. ccw-vscode-bridge/.vscodeignore - 打包排除文件
5. ccw-vscode-bridge/README.md - 使用文档

待创建的文件:

1. ccw/src/tools/vscode-lsp.ts - MCP工具实现
2. ccw/src/tools/index.ts - 注册新工具

A.3 VSCode Bridge Extension Implementation

File: ccw-vscode-bridge/src/extension.ts

// 核心功能:
// 1. 启动HTTP服务器监听3457端口
// 2. 接收POST请求，解析JSON body
// 3. 调用VSCode内置LSP命令
// 4. 返回JSON结果

// HTTP Endpoints:
// POST /get_definition    → vscode.executeDefinitionProvider
// POST /get_references    → vscode.executeReferenceProvider
// POST /get_hover         → vscode.executeHoverProvider
// POST /get_document_symbols → vscode.executeDocumentSymbolProvider

A.4 MCP Tool Implementation

File: ccw/src/tools/vscode-lsp.ts

/**
 * MCP tool that communicates with VSCode Bridge extension.
 * 
 * Actions:
 * - get_definition: Find symbol definition
 * - get_references: Find all references
 * - get_hover: Get hover information
 * - get_document_symbols: List symbols in file
 * 
 * Required:
 * - ccw-vscode-bridge extension running in VSCode
 * - File must be open in VSCode for accurate results
 */

const schema: ToolSchema = {
  name: 'vscode_lsp',
  description: `Access live VSCode LSP features...`,
  inputSchema: {
    type: 'object',
    properties: {
      action: { type: 'string', enum: [...] },
      file_path: { type: 'string' },
      line: { type: 'number' },
      character: { type: 'number' }
    },
    required: ['action', 'file_path']
  }
};

A.5 Advantages vs Standalone LSP Server

Feature	VSCode Bridge	Standalone LSP Server
Setup Complexity	Low (VSCode ext)	Medium (multiple LS)
Language Support	Automatic (VSCode)	Manual config
Maintenance	Low	Medium
IDE Independence	VSCode only	Any LSP client
Performance	Good	Good
Advanced Features	Full VSCode support	LSP standard

---

Appendix B: Complete Integration Architecture

B.1 Three Integration Paths

┌─────────────────────────────────────────────────────────────────────────────┐
│                          CodexLens Integration Paths                         │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  Path 1: VSCode Bridge (HTTP)          Path 2: Standalone LSP Server        │
│  ────────────────────────              ─────────────────────────────        │
│                                                                              │
│  ┌─────────────┐                       ┌─────────────┐                      │
│  │ CCW MCP     │                       │  Any LSP    │                      │
│  │ vscode_lsp  │                       │   Client    │                      │
│  └──────┬──────┘                       └──────┬──────┘                      │
│         │ HTTP                                │ LSP/stdio                    │
│         ▼                                     ▼                              │
│  ┌─────────────┐                       ┌─────────────┐                      │
│  │ ccw-vscode  │                       │ codexlens-  │                      │
│  │   -bridge   │                       │    lsp      │                      │
│  └──────┬──────┘                       └──────┬──────┘                      │
│         │ VSCode API                          │ Child Process               │
│         ▼                                     ▼                              │
│  ┌─────────────┐                       ┌─────────────┐                      │
│  │   VSCode    │                       │   pylsp     │                      │
│  │     LS      │                       │  tsserver   │                      │
│  └─────────────┘                       │   gopls     │                      │
│                                        └─────────────┘                      │
│                                                                              │
│                       Path 3: Index-Based (Current)                         │
│                       ─────────────────────────────                         │
│                                                                              │
│                       ┌─────────────┐                                       │
│                       │ CCW MCP     │                                       │
│                       │codex_lens_lsp│                                      │
│                       └──────┬──────┘                                       │
│                              │ Python subprocess                            │
│                              ▼                                              │
│                       ┌─────────────┐                                       │
│                       │ CodexLens   │                                       │
│                       │   Index DB  │                                       │
│                       └─────────────┘                                       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

B.2 Recommendation Matrix

Use Case	Recommended Path	Reason
Claude Code + VSCode	Path 1: VSCode Bridge	Simplest, full VSCode features
CLI-only workflows	Path 2: Standalone LSP	No VSCode dependency
Quick search across indexed code	Path 3: Index-based	Fastest response
Multi-IDE support	Path 2: Standalone LSP	Standard protocol
Advanced refactoring	Path 1: VSCode Bridge	Full VSCode capabilities

B.3 Hybrid Mode (Recommended)

For maximum flexibility, implement all three paths:

// Smart routing in CCW
function selectLSPPath(request) {
  // 1. Try VSCode Bridge first (if available)
  if (await checkVSCodeBridge()) {
    return "vscode_bridge";
  }
  
  // 2. Fall back to Standalone LSP
  if (await checkStandaloneLSP(request.fileType)) {
    return "standalone_lsp";
  }
  
  // 3. Last resort: Index-based
  return "index_based";
}

---

Appendix C: Implementation Tasks Summary

C.1 VSCode Bridge Tasks

Task ID	Description	Priority	Status
VB-1	Create ccw-vscode-bridge extension structure	High	✅ Done
VB-2	Implement HTTP server in extension.ts	High	✅ Done
VB-3	Create vscode_lsp MCP tool	High	🔄 Pending
VB-4	Register tool in CCW	High	🔄 Pending
VB-5	Test with VSCode	Medium	🔄 Pending
VB-6	Add connection retry logic	Low	🔄 Pending

C.2 Standalone LSP Server Tasks

Task ID	Description	Priority	Status
LSP-1	Setup pygls project structure	High	🔄 Pending
LSP-2	Implement multiplexer	High	🔄 Pending
LSP-3	Core handlers (definition, references)	High	🔄 Pending
LSP-4	Position tolerance	Medium	🔄 Pending
LSP-5	Tests and documentation	Medium	🔄 Pending

C.3 Integration Tasks

Task ID	Description	Priority	Status
INT-1	Smart path routing	Medium	🔄 Pending
INT-2	Unified error handling	Medium	🔄 Pending
INT-3	Performance benchmarks	Low	🔄 Pending

---

Questions for Clarification

Before implementation, confirm:

1. Implementation Priority: Start with VSCode Bridge (simpler) or Standalone LSP (more general)?
2. Language Priority: Which languages are most important? (TypeScript, Python, Go, Rust, etc.)
3. IDE Focus: Target VS Code first, then others?
4. Fallback Strategy: Should we keep index-based search as fallback if LSP fails?
5. Caching: How much should we cache LS responses?
6. Configuration: Simple JSON config or more sophisticated format?