diff --git a/.claude/.workflow-enhancements/test-enrichment-integration.md b/.claude/.workflow-enhancements/test-enrichment-integration.md new file mode 100644 index 00000000..904a2e48 --- /dev/null +++ b/.claude/.workflow-enhancements/test-enrichment-integration.md @@ -0,0 +1,216 @@ +# Test Generation Enhancement - Implementation Summary + +## Overview + +Enhanced test-task-generate workflow with two major improvements: +1. **Gemini Test Enhancement** (Phase 1.5): Comprehensive test suggestions for API, integration, and error scenarios +2. **Planning Notes Mechanism** (test-planning-notes.md): Record Gemini enhancement, context, and plan process for full transparency + +--- + +## Changes Made + +### 1. Created Gemini Prompt Template +**File**: `~/.claude/workflows/cli-templates/prompts/test/test-suggestions-enhancement.txt` +- Structured prompt for Gemini CLI to generate comprehensive test suggestions +- Covers L1-L3 test layers with focus on API, integration, and error scenarios +- Produces actionable test specifications with expected behavior and success criteria + +### 2. Enhanced test-task-generate.md with Two Major Features + +**Location**: `.claude/commands/workflow/tools/test-task-generate.md` + +#### A. Phase 1.5: Gemini Test Enhancement +- **Invocation of cli-execution-agent** for Gemini analysis +- **Template**: test-suggestions-enhancement.txt +- **Output**: gemini-enriched-suggestions.md +- **Merge**: Appends enriched suggestions to TEST_ANALYSIS_RESULTS.md + +#### B. test-planning-notes.md Mechanism (NEW - References plan.md pattern) +Similar to plan.md's planning-notes.md, this records the entire planning journey. + +**Lifecycle**: +1. **Phase 1 Creation**: Initialize test-planning-notes.md with Test Intent +2. **Phase 1.5 Update**: Record Gemini Enhancement findings +3. **Phase 2 Update**: Final Task Generation findings (agent-executed) + +--- + +## Planning Notes Architecture (Similar to plan.md) + +### test-planning-notes.md Structure + +```markdown +# Test Planning Notes + +**Session**: WFS-test-xxx +**Source Session**: WFS-xxx +**Created**: [timestamp] + +## Test Intent (Phase 1) +- **PROJECT_TYPE**: [Detected project type] +- **TEST_FRAMEWORK**: [Detected framework] +- **COVERAGE_TARGET**: [Coverage goal] +- **SOURCE_SESSION**: [Source session ID] + +--- + +## Context Findings (Phase 1) +### Files with Coverage Gaps +[Extracted from TEST_ANALYSIS_RESULTS.md] + +### Test Framework & Conventions +- Framework: [Framework] +- Coverage Target: [Target] + +--- + +## Gemini Enhancement (Phase 1.5) + +**Analysis Timestamp**: [timestamp] +**Template**: test-suggestions-enhancement.txt +**Output File**: .process/gemini-enriched-suggestions.md + +### Enriched Test Suggestions (Complete Gemini Analysis) + +[COMPLETE CONTENT FROM gemini-enriched-suggestions.md EMBEDDED HERE] +[All L1-L3 test suggestions, API contracts, integration patterns, error scenarios] + +### Gemini Analysis Summary +- **Status**: Enrichment complete +- **Layers Covered**: L1, L2.1, L2.2, L2.4, L2.5 +- **Focus Areas**: API contracts, integration patterns, error scenarios, edge cases +- **Output Stored**: Full analysis in gemini-enriched-suggestions.md + +### Key Features + +1. **Complete Gemini Analysis Embedded**: Full enriched suggestions content is directly embedded into test-planning-notes.md +2. **Preserved Output File**: gemini-enriched-suggestions.md still generated as artifact for reference +3. **Consolidated Context**: test-planning-notes.md becomes single source of truth for all planning context +4. **Agent Friendly**: test-action-planning-agent can read complete planning context from one file + +### Use Cases for Gemini Content in planning-notes.md + +| Use Case | Benefit | +|----------|---------| +| Reviewer reads planning-notes | See full Gemini suggestions without opening separate file | +| Agent reads planning-notes | Get enriched context alongside other planning metadata | +| Archive session | Complete record of analysis in one file | +| N+1 phase reference | Full history of suggestions for iterative refinement | + +--- + +## Consolidated Test Requirements (Phase 2 Input) +1. [Context] Test framework conventions +2. [Context] Coverage target +3. [Gemini] API contract test strategies +4. [Gemini] Error scenario coverage +[... more consolidated requirements] + +--- + +## Task Generation (Phase 2) +[Updated by test-action-planning-agent] + +## N+1 Context +### Decisions +| Decision | Rationale | Revisit? | +|----------|-----------|----------| +[Decisions made during planning] + +### Deferred +- [ ] Items for N+1 phase +``` + +### Key Benefits of Planning Notes + +1. **Full Transparency**: Record Gemini enhancement findings and context +2. **Consolidated Context**: All phase inputs/outputs in one place +3. **N+1 Support**: Track decisions and deferred items for future iterations +4. **Agent Context**: test-action-planning-agent reads consolidated requirements +5. **Traceability**: How each test requirement came from which phase/source +6. **Integration Consistency**: Follows same pattern as plan.md for familiarity + +--- + +## Complete Integration Flow + +``` +test-task-generate execution: + ├─ Phase 1: Context Preparation + │ ├─ Load session metadata + │ ├─ Load TEST_ANALYSIS_RESULTS.md (original) + │ └─ CREATE test-planning-notes.md (Test Intent section) + │ + ├─ Phase 1.5: Gemini Test Enhancement + │ ├─ Invoke cli-execution-agent for Gemini analysis + │ ├─ Generate gemini-enriched-suggestions.md (full analysis output) + │ └─ RECORD complete enrichment to test-planning-notes.md (embed full Gemini analysis) + │ + └─ Phase 2: Test Document Generation (Agent) + ├─ Load TEST_ANALYSIS_RESULTS.md (original, unchanged) + ├─ Load test-planning-notes.md (includes full Gemini enrichment) + ├─ Generate task JSONs (IMPL-*.json) + └─ Create IMPL_PLAN.md + TODO_LIST.md +``` + +## Key Design Decisions + +### Gemini Enhancement Recording Strategy + +**Decision**: Record Gemini enhancement to test-planning-notes.md instead of appending to TEST_ANALYSIS_RESULTS.md + +**Rationale**: +1. **Separation of Concerns**: TEST_ANALYSIS_RESULTS.md remains as original analysis output +2. **Single Source of Truth**: test-planning-notes.md consolidates ALL planning context +3. **Better Traceability**: Clear separation between base analysis and AI enrichment +4. **Follows plan.md Pattern**: Consistent with planning-notes.md mechanism + +**Implementation**: +- Phase 1.5 generates gemini-enriched-suggestions.md (preserved as output file) +- Complete Gemini analysis content is embedded into test-planning-notes.md +- test-action-planning-agent reads both TEST_ANALYSIS_RESULTS.md and test-planning-notes.md + +## No Breaking Changes + +**Backward Compatible**: +- Phase 1.5 is optional enhancement - if Gemini analysis fails, workflow continues with original TEST_ANALYSIS_RESULTS.md +- test-action-planning-agent requires NO modifications to source code +- test-planning-notes.md consolidates context but doesn't replace existing inputs +- Existing workflows unaffected; only adds richer suggestions and traceability + +**Enriched Layers**: +- **L1 (Unit)**: Edge cases, boundary conditions, error path validation +- **L2.1 (Integration)**: Module interactions, dependency injection patterns +- **L2.2 (API Contracts)**: Request/response, validation, status codes, error responses +- **L2.4 (External APIs)**: Mock strategies, failure scenarios, timeout handling +- **L2.5 (Failure Modes)**: Exception hierarchies, error propagation, recovery strategies + +**Benefits**: +- ✅ Comprehensive API test specifications +- ✅ Detailed integration test strategies +- ✅ Systematic error scenario coverage +- ✅ More actionable task generation +- ✅ Reduced manual test planning effort + +## How Test-Action-Planning-Agent Uses Enrichment + +**Phase 1 (Context Loading)**: +- test-action-planning-agent reads complete TEST_ANALYSIS_RESULTS.md +- Includes both original analysis AND Gemini-enhanced suggestions +- Automatically leverages enriched content for task generation + +**IMPL-001.json (Test Generation)**: +- Generated with reference to enriched test specifications +- Enhanced L1-L3 layer requirements from Gemini analysis +- More detailed test case descriptions +- Better integration and error scenario coverage + +## Verification + +To verify the enhancement is working: +1. Run `/workflow:test-fix-gen` to trigger test workflow +2. Check `.workflow/active/[session]/.process/gemini-enriched-suggestions.md` exists +3. Check TEST_ANALYSIS_RESULTS.md contains "## Gemini-Enhanced Test Suggestions" section +4. Review IMPL-001.json - should reference enriched test specifications +5. Execute `/workflow:test-cycle-execute` - generated tests should cover API, integration, and error scenarios comprehensively diff --git a/.claude/commands/workflow/tools/test-task-generate.md b/.claude/commands/workflow/tools/test-task-generate.md index 8619d065..fa25dfb3 100644 --- a/.claude/commands/workflow/tools/test-task-generate.md +++ b/.claude/commands/workflow/tools/test-task-generate.md @@ -38,17 +38,209 @@ Phase 1: Context Preparation (Command) │ ├─ session_metadata_path │ ├─ test_analysis_results_path (REQUIRED) │ └─ test_context_package_path - └─ Provide metadata (session_id, source_session_id) + ├─ Provide metadata (session_id, source_session_id) + └─ Create test-planning-notes.md (User Intent section) + +Phase 1.5: Gemini Test Enhancement (Command) + ├─ Invoke cli-execution-agent for Gemini analysis + ├─ Read TEST_ANALYSIS_RESULTS.md for context + ├─ Generate enriched test suggestions (API, integration, error scenarios) + └─ Record enriched suggestions to test-planning-notes.md (Gemini Enhancement section) Phase 2: Test Document Generation (Agent) - ├─ Load TEST_ANALYSIS_RESULTS.md as primary requirements source + ├─ Load TEST_ANALYSIS_RESULTS.md (with enriched suggestions) + ├─ Load test-planning-notes.md (consolidated context) ├─ Generate Test Task JSON Files (.task/IMPL-*.json) │ ├─ IMPL-001: Test generation (L1-L3 layers, project-specific templates) │ ├─ IMPL-001.3: Code validation gate (L0 + AI issue detection) │ ├─ IMPL-001.5: Test quality gate (anti-patterns + coverage) │ └─ IMPL-002: Test execution & fix cycle ├─ Create IMPL_PLAN.md (test_session variant) - └─ Generate TODO_LIST.md with test phase indicators + ├─ Generate TODO_LIST.md with test phase indicators + └─ Update test-planning-notes.md (Task Generation section) +``` + +--- + +## Phase 1: Context Preparation + +**Purpose**: Assemble test session paths, load test analysis context, and create test-planning-notes.md. + +**Execution Steps**: +1. Parse `--session` flag to get test session ID +2. Load `workflow-session.json` for session metadata +3. Verify `TEST_ANALYSIS_RESULTS.md` exists (from test-concept-enhanced) +4. Load `test-context-package.json` for coverage data +5. Create `test-planning-notes.md` with initial context + +**After Phase 1**: Initialize test-planning-notes.md + +```javascript +// Create test-planning-notes.md with N+1 context support +const testPlanningNotesPath = `.workflow/active/${testSessionId}/test-planning-notes.md` +const sessionMetadata = JSON.parse(Read(`.workflow/active/${testSessionId}/workflow-session.json`)) +const testAnalysis = Read(`.workflow/active/${testSessionId}/.process/TEST_ANALYSIS_RESULTS.md`) +const sourceSessionId = sessionMetadata.source_session_id || 'N/A' + +// Extract key info from TEST_ANALYSIS_RESULTS.md +const projectType = testAnalysis.match(/Project Type:\s*(.+)/)?.[1] || 'Unknown' +const testFramework = testAnalysis.match(/Test Framework:\s*(.+)/)?.[1] || 'Unknown' +const coverageTarget = testAnalysis.match(/Coverage Target:\s*(.+)/)?.[1] || '80%' + +Write(testPlanningNotesPath, `# Test Planning Notes + +**Session**: ${testSessionId} +**Source Session**: ${sourceSessionId} +**Created**: ${new Date().toISOString()} + +## Test Intent (Phase 1) + +- **PROJECT_TYPE**: ${projectType} +- **TEST_FRAMEWORK**: ${testFramework} +- **COVERAGE_TARGET**: ${coverageTarget} +- **SOURCE_SESSION**: ${sourceSessionId} + +--- + +## Context Findings (Phase 1) + +### Files with Coverage Gaps +(Extracted from TEST_ANALYSIS_RESULTS.md) + +### Test Framework & Conventions +- Framework: ${testFramework} +- Coverage Target: ${coverageTarget} + +--- + +## Gemini Enhancement (Phase 1.5) +(To be filled by Gemini analysis) + +### Enhanced Test Suggestions +- **L1 (Unit)**: (Pending) +- **L2.1 (Integration)**: (Pending) +- **L2.2 (API Contracts)**: (Pending) +- **L2.4 (External APIs)**: (Pending) +- **L2.5 (Failure Modes)**: (Pending) + +### Gemini Analysis Summary +(Pending enrichment) + +--- + +## Consolidated Test Requirements (Phase 2 Input) +1. [Context] ${testFramework} framework conventions +2. [Context] ${coverageTarget} coverage target + +--- + +## Task Generation (Phase 2) +(To be filled by test-action-planning-agent) + +## N+1 Context +### Decisions +| Decision | Rationale | Revisit? | +|----------|-----------|----------| + +### Deferred +- [ ] (For N+1) +`) + +--- + +## Phase 1.5: Gemini Test Enhancement + +**Purpose**: Enrich test specifications with comprehensive test suggestions and record to test-planning-notes.md. + +**Execution Steps**: +1. Load TEST_ANALYSIS_RESULTS.md from `.workflow/active/{test-session-id}/.process/` +2. Invoke `cli-execution-agent` with Gemini for test enhancement analysis +3. Use template: `~/.claude/workflows/cli-templates/prompts/test/test-suggestions-enhancement.txt` +4. Gemini generates enriched test suggestions across L1-L3 layers → gemini-enriched-suggestions.md +5. Record enriched suggestions to test-planning-notes.md (Gemini Enhancement section) + +**Agent Invocation**: +```javascript +Task( + subagent_type="cli-execution-agent", + run_in_background=false, + description="Enhance test specifications with Gemini analysis", + prompt=` +## Task Objective +Analyze TEST_ANALYSIS_RESULTS.md and generate enriched test suggestions using Gemini CLI + +## Input Files +- Read: .workflow/active/{test-session-id}/.process/TEST_ANALYSIS_RESULTS.md +- Extract: Project type, test framework, coverage gaps, identified files + +## Gemini Analysis Execution +Execute Gemini with comprehensive test enhancement prompt: + ccw cli -p "[comprehensive test prompt]" --tool gemini --mode analysis --rule analysis-test-strategy-enhancement --cd .workflow/active/{test-session-id}/.process + +## Expected Output +Generate gemini-enriched-suggestions.md with structured test enhancements: + - L1 (Unit Tests): Edge cases, boundaries, error paths + - L2.1 (Integration): Module interactions, dependency injection + - L2.2 (API Contracts): Request/response, validation, error responses + - L2.4 (External APIs): Mock strategies, failure scenarios, timeouts + - L2.5 (Failure Modes): Exception handling, error propagation, recovery + +## Validation +- gemini-enriched-suggestions.md created and complete +- Suggestions are actionable and specific (not generic) +- All L1-L3 layers covered +` +) +``` + +**Output**: gemini-enriched-suggestions.md (complete Gemini analysis) + +**After Phase 1.5**: Update test-planning-notes.md with Gemini enhancement findings + +```javascript +// Read enriched suggestions from gemini-enriched-suggestions.md +const enrichedSuggestionsPath = `.workflow/active/${testSessionId}/.process/gemini-enriched-suggestions.md` +const enrichedSuggestions = Read(enrichedSuggestionsPath) + +// Update Phase 1.5 section in test-planning-notes.md with full enriched suggestions +Edit(testPlanningNotesPath, { + old: '## Gemini Enhancement (Phase 1.5)\n(To be filled by Gemini analysis)\n\n### Enhanced Test Suggestions\n- **L1 (Unit)**: (Pending)\n- **L2.1 (Integration)**: (Pending)\n- **L2.2 (API Contracts)**: (Pending)\n- **L2.4 (External APIs)**: (Pending)\n- **L2.5 (Failure Modes)**: (Pending)\n\n### Gemini Analysis Summary\n(Pending enrichment)', + new: `## Gemini Enhancement (Phase 1.5) + +**Analysis Timestamp**: ${new Date().toISOString()} +**Template**: test-suggestions-enhancement.txt +**Output File**: .process/gemini-enriched-suggestions.md + +### Enriched Test Suggestions (Complete Gemini Analysis) + +${enrichedSuggestions} + +### Gemini Analysis Summary +- **Status**: Enrichment complete +- **Layers Covered**: L1, L2.1, L2.2, L2.4, L2.5 +- **Focus Areas**: API contracts, integration patterns, error scenarios, edge cases +- **Output Stored**: Full analysis in gemini-enriched-suggestions.md` +}) + +// Append Gemini constraints to consolidated test requirements +const geminiConstraints = [ + '[Gemini] Implement all suggested L1 edge cases and boundary tests', + '[Gemini] Apply L2.1 module interaction patterns from analysis', + '[Gemini] Follow L2.2 API contract test matrix from analysis', + '[Gemini] Use L2.4 external API mock strategies from analysis', + '[Gemini] Cover L2.5 error scenarios from analysis' +] + +const currentNotes = Read(testPlanningNotesPath) +const constraintCount = (currentNotes.match(/^\d+\./gm) || []).length + +Edit(testPlanningNotesPath, { + old: '## Consolidated Test Requirements (Phase 2 Input)', + new: `## Consolidated Test Requirements (Phase 2 Input) +1. [Context] ${testFramework} framework conventions +2. [Context] ${coverageTarget} coverage target +${geminiConstraints.map((c, i) => `${i + 3}. ${c}`).join('\n')}` +}) ``` --- @@ -70,8 +262,10 @@ IMPORTANT: This is TEST PLANNING ONLY - you are generating planning documents, N Input: - Session Metadata: .workflow/active/{test-session-id}/workflow-session.json - TEST_ANALYSIS_RESULTS: .workflow/active/{test-session-id}/.process/TEST_ANALYSIS_RESULTS.md (REQUIRED) + - Test Planning Notes: .workflow/active/{test-session-id}/test-planning-notes.md (REQUIRED - contains Gemini enhancement findings) - Test Context Package: .workflow/active/{test-session-id}/.process/test-context-package.json - Context Package: .workflow/active/{test-session-id}/.process/context-package.json + - Enriched Suggestions: .workflow/active/{test-session-id}/.process/gemini-enriched-suggestions.md (for reference) - Source Session Summaries: .workflow/active/{source-session-id}/.summaries/IMPL-*.md (if exists) Output: @@ -85,6 +279,15 @@ Workflow Type: test_session Source Session: {source-session-id} (if exists) MCP Capabilities: {exa_code, exa_web, code_index} +## CONSOLIDATED CONTEXT +**From test-planning-notes.md**: +- Test Intent: Project type, test framework, coverage target +- Context Findings: Coverage gaps, file analysis +- Gemini Enhancement: Complete enriched test suggestions (L1-L3 layers) + * Full analysis embedded in planning-notes.md + * API contracts, integration patterns, error scenarios +- Consolidated Requirements: Combined constraints from all phases + ## YOUR SPECIFICATIONS You are @test-action-planning-agent. Your complete test specifications are defined in: d:\Claude_dms3\.claude\agents\test-action-planning-agent.md @@ -148,6 +351,7 @@ This includes: ├── workflow-session.json # Session metadata ├── IMPL_PLAN.md # Test implementation plan ├── TODO_LIST.md # Task checklist +├── test-planning-notes.md # [NEW] Consolidated planning notes with full Gemini analysis ├── .task/ │ ├── IMPL-001.json # Test generation (L1-L3) │ ├── IMPL-001.3-validation.json # Code validation gate (L0 + AI) @@ -155,7 +359,8 @@ This includes: │ └── IMPL-002.json # Test execution & fix cycle └── .process/ ├── test-context-package.json # Test coverage and patterns - └── TEST_ANALYSIS_RESULTS.md # L0-L3 requirements (source for agent) + ├── gemini-enriched-suggestions.md # [NEW] Gemini-generated test enhancements + └── TEST_ANALYSIS_RESULTS.md # L0-L3 requirements (original from test-concept-enhanced) ``` ### Task Summary diff --git a/.claude/workflows/cli-templates/prompts/test/test-suggestions-enhancement.txt b/.claude/workflows/cli-templates/prompts/test/test-suggestions-enhancement.txt new file mode 100644 index 00000000..59951b6d --- /dev/null +++ b/.claude/workflows/cli-templates/prompts/test/test-suggestions-enhancement.txt @@ -0,0 +1,38 @@ +PURPOSE: Generate comprehensive multi-layer test enhancement suggestions + - Success: Cover L0-L3 layers with focus on API, integration, and error scenarios + - Scope: Files with coverage gaps identified in TEST_ANALYSIS_RESULTS.md + - Goal: Provide specific, actionable test case suggestions that increase coverage completeness + +TASK: + • L1 (Unit Tests): Suggest edge cases, boundary conditions, error paths, state transitions + • L2.1 (Integration): Suggest module interaction patterns, dependency injection scenarios + • L2.2 (API Contracts): Suggest request/response test cases, validation, status codes, error responses + • L2.4 (External APIs): Suggest mock strategies, failure scenarios, timeout handling, retry logic + • L2.5 (Failure Modes): Suggest exception hierarchies, error propagation, recovery strategies + • Cross-cutting: Suggest performance test cases, security considerations + +MODE: analysis + +CONTEXT: @.workflow/active/{test-session-id}/.process/TEST_ANALYSIS_RESULTS.md + Memory: Project type, test framework, existing test patterns, coverage gaps + +EXPECTED: Markdown report with structured test enhancement suggestions organized by: + 1. File-level test requirements (per file needing tests) + 2. Layer-specific test cases (L1, L2.1, L2.2, L2.4, L2.5) + 3. Each suggestion includes: + - Test type and layer (e.g., "L2.2 API Contract Test") + - Specific test case description (e.g., "POST /api/users - Invalid email format") + - Expected behavior (e.g., "Returns 400 with validation error message") + - Dependencies/mocks needed (e.g., "Mock email service") + - Success criteria (e.g., "Status 400, error.field === 'email'") + 4. Test ordering/dependencies (which tests should run first) + 5. Integration test strategies (how components interact) + 6. Error scenario matrix (all failure modes covered) + +CONSTRAINTS: + - Focus on identified coverage gaps from TEST_ANALYSIS_RESULTS.md + - Prioritize API tests, integration tests, and error scenarios + - No code generation - suggestions only with sufficient detail for implementation + - Consider project conventions and existing test patterns + - Each suggestion should be actionable and specific (not generic) + - Output format: Markdown with clear section headers