From 1dd7b2394280a31c1d1d7dc759f95c55ddfe07e4 Mon Sep 17 00:00:00 2001 From: cexll Date: Sat, 14 Feb 2026 23:58:38 +0800 Subject: [PATCH] feat harness skill --- skills/harness/SKILL.md | 329 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 329 insertions(+) create mode 100644 skills/harness/SKILL.md diff --git a/skills/harness/SKILL.md b/skills/harness/SKILL.md new file mode 100644 index 0000000..3ae40cf --- /dev/null +++ b/skills/harness/SKILL.md @@ -0,0 +1,329 @@ +--- +name: harness +description: "This skill should be used for multi-session autonomous agent work requiring progress checkpointing, failure recovery, and task dependency management. Triggers on '/harness' command, or when a task involves many subtasks needing progress persistence, sleep/resume cycles across context windows, recovery from mid-task failures with partial state, or distributed work across multiple agent sessions. Synthesized from Anthropic and OpenAI engineering practices for long-running agents." +--- + +# Harness — Long-Running Agent Framework + +Executable protocol enabling any agent task to run continuously across multiple sessions with automatic progress recovery, task dependency resolution, failure rollback, and standardized error handling. + +## Design Principles + +1. **Design for the agent, not the human** — Test output, docs, and task structure are the agent's primary interface +2. **Progress files ARE the context** — When context window resets, progress files + git history = full recovery +3. **Premature completion is the #1 failure mode** — Structured task lists with explicit completion criteria prevent declaring victory early +4. **Standardize everything grep-able** — ERROR on same line, structured timestamps, consistent prefixes +5. **Fast feedback loops** — Pre-compute stats, run smoke tests before full validation +6. **Idempotent everything** — Init scripts, task execution, environment setup must all be safe to re-run +7. **Fail safe, not fail silent** — Every failure must have an explicit recovery strategy + +## Commands + +``` +/harness init # Initialize harness files in project +/harness run # Start/resume the infinite loop +/harness status # Show current progress and stats +/harness add "task description" # Add a task to the list +``` + +## Progress Persistence (Dual-File System) + +Maintain two files in the project working directory: + +### harness-progress.txt (Append-Only Log) + +Free-text log of all agent actions across sessions. Never truncate. + +``` +[2025-07-01T10:00:00Z] [SESSION-1] INIT Harness initialized for project /path/to/project +[2025-07-01T10:00:05Z] [SESSION-1] INIT Environment health check: PASS +[2025-07-01T10:00:10Z] [SESSION-1] LOCK acquired (pid=12345) +[2025-07-01T10:00:11Z] [SESSION-1] Starting [task-001] Implement user authentication (base=def5678) +[2025-07-01T10:05:00Z] [SESSION-1] CHECKPOINT [task-001] step=2/4 "auth routes created, tests pending" +[2025-07-01T10:15:30Z] [SESSION-1] Completed [task-001] (commit abc1234) +[2025-07-01T10:15:31Z] [SESSION-1] Starting [task-002] Add rate limiting (base=abc1234) +[2025-07-01T10:20:00Z] [SESSION-1] ERROR [task-002] [TASK_EXEC] Redis connection refused +[2025-07-01T10:20:01Z] [SESSION-1] ROLLBACK [task-002] git reset --hard abc1234 +[2025-07-01T10:20:02Z] [SESSION-1] STATS tasks_total=5 completed=1 failed=1 pending=3 blocked=0 attempts_total=2 checkpoints=1 +``` + +### harness-tasks.json (Structured State) + +```json +{ + "version": 2, + "created": "2025-07-01T10:00:00Z", + "session_config": { + "max_tasks_per_session": 20, + "max_sessions": 50 + }, + "tasks": [ + { + "id": "task-001", + "title": "Implement user authentication", + "status": "completed", + "priority": "P0", + "depends_on": [], + "attempts": 1, + "max_attempts": 3, + "started_at_commit": "def5678", + "validation": { + "command": "npm test -- --testPathPattern=auth", + "timeout_seconds": 300 + }, + "on_failure": { + "cleanup": null + }, + "error_log": [], + "checkpoints": [], + "completed_at": "2025-07-01T10:15:30Z" + }, + { + "id": "task-002", + "title": "Add rate limiting", + "status": "failed", + "priority": "P1", + "depends_on": [], + "attempts": 1, + "max_attempts": 3, + "started_at_commit": "abc1234", + "validation": { + "command": "npm test -- --testPathPattern=rate-limit", + "timeout_seconds": 120 + }, + "on_failure": { + "cleanup": "docker compose down redis" + }, + "error_log": ["[TASK_EXEC] Redis connection refused"], + "checkpoints": [], + "completed_at": null + }, + { + "id": "task-003", + "title": "Add OAuth providers", + "status": "pending", + "priority": "P1", + "depends_on": ["task-001"], + "attempts": 0, + "max_attempts": 3, + "started_at_commit": null, + "validation": { + "command": "npm test -- --testPathPattern=oauth", + "timeout_seconds": 180 + }, + "on_failure": { + "cleanup": null + }, + "error_log": [], + "checkpoints": [], + "completed_at": null + } + ], + "session_count": 1, + "last_session": "2025-07-01T10:20:02Z" +} +``` + +Task statuses: `pending` → `in_progress` (transient, set only during active execution) → `completed` or `failed`. A task found as `in_progress` at session start means the previous session was interrupted — handle via Context Window Recovery Protocol. + +**Session boundary**: A session starts when the agent begins executing the Session Start protocol and ends when a Stopping Condition is met or the context window resets. Each session gets a unique `SESSION-N` identifier (N = `session_count` after increment). + +## Concurrency Control + +Before modifying `harness-tasks.json`, acquire an exclusive lock using portable `mkdir` (atomic on all POSIX systems, works on both macOS and Linux): + +```bash +# Acquire lock (fail fast if another agent is running) +LOCKDIR="/tmp/harness-$(printf '%s' "$(pwd)" | shasum -a 256 2>/dev/null || sha256sum | cut -c1-8).lock" +if ! mkdir "$LOCKDIR" 2>/dev/null; then + # Check if lock holder is still alive + LOCK_PID=$(cat "$LOCKDIR/pid" 2>/dev/null) + if [ -n "$LOCK_PID" ] && kill -0 "$LOCK_PID" 2>/dev/null; then + echo "ERROR: Another harness session is active (pid=$LOCK_PID)"; exit 1 + fi + # Stale lock — atomically reclaim via mv to avoid TOCTOU race + STALE="$LOCKDIR.stale.$$" + if mv "$LOCKDIR" "$STALE" 2>/dev/null; then + rm -rf "$STALE" + mkdir "$LOCKDIR" || { echo "ERROR: Lock contention"; exit 1; } + echo "WARN: Removed stale lock${LOCK_PID:+ from pid=$LOCK_PID}" + else + echo "ERROR: Another agent reclaimed the lock"; exit 1 + fi +fi +echo "$$" > "$LOCKDIR/pid" +trap 'rm -rf "$LOCKDIR"' EXIT +``` + +Log lock acquisition: `[timestamp] [SESSION-N] LOCK acquired (pid=)` +Log lock release: `[timestamp] [SESSION-N] LOCK released` + +The lock is held for the entire session. The `trap EXIT` handler releases it automatically on normal exit, errors, or signals. Never release the lock between tasks within a session. + +## Infinite Loop Protocol + +### Session Start (Execute Every Time) + +1. **Read state**: Read last 200 lines of `harness-progress.txt` + full `harness-tasks.json`. If JSON is unparseable, see JSON corruption recovery in Error Handling. +2. **Read git**: Run `git log --oneline -20` and `git diff --stat` to detect uncommitted work +3. **Acquire lock**: Fail if another session is active +4. **Recover interrupted tasks** (see Context Window Recovery below) +5. **Health check**: Run `harness-init.sh` if it exists +6. **Track session**: Increment `session_count` in JSON. Check `session_count` against `max_sessions` — if reached, log STATS and STOP. Initialize per-session task counter to 0. +7. **Pick next task** using Task Selection Algorithm below + +### Task Selection Algorithm + +Before selecting, run dependency validation: + +1. **Cycle detection**: For each non-completed task, walk `depends_on` transitively. If any task appears in its own chain, mark it `failed` with `[DEPENDENCY] Circular dependency detected: task-A -> task-B -> task-A`. Self-references (`depends_on` includes own id) are also cycles. +2. **Blocked propagation**: If a task's `depends_on` includes a task that is `failed` and will never be retried (either `attempts >= max_attempts` OR its `error_log` contains a `[DEPENDENCY]` entry), mark the blocked task as `failed` with `[DEPENDENCY] Blocked by failed task-XXX`. Repeat until no more tasks can be propagated. + +Then pick the next task in this priority order: + +1. Tasks with `status: "pending"` where ALL `depends_on` tasks are `completed` — sorted by `priority` (P0 > P1 > P2), then by `id` (lowest first) +2. Tasks with `status: "failed"` where `attempts < max_attempts` and ALL `depends_on` are `completed` — sorted by priority, then oldest failure first +3. If no eligible tasks remain → log final STATS → STOP + +### Task Execution Cycle + +For each task, execute this exact sequence: + +1. **Claim**: Record `started_at_commit` = current HEAD hash. Set status to `in_progress`, log `Starting [] (base=<hash>)` +2. **Execute with checkpoints**: Perform the work. After each significant step, log: + ``` + [timestamp] [SESSION-N] CHECKPOINT [task-id] step=M/N "description of what was done" + ``` + Also append to the task's `checkpoints` array: `{ "step": M, "total": N, "description": "...", "timestamp": "ISO" }` +3. **Validate**: Run the task's `validation.command` wrapped with `timeout`: `timeout <timeout_seconds> <command>`. If no validation command, skip. Before running, verify the command exists (e.g., `command -v <binary>`) — if missing, treat as `ENV_SETUP` error. + - Command exits 0 → PASS + - Command exits non-zero → FAIL + - Command exceeds timeout → TIMEOUT +4. **Record outcome**: + - **Success**: status=`completed`, set `completed_at`, log `Completed [<task-id>] (commit <hash>)`, git commit + - **Failure**: increment `attempts`, append error to `error_log`. Verify `started_at_commit` exists via `git cat-file -t <hash>` — if missing, mark failed at max_attempts. Otherwise execute `git reset --hard <started_at_commit>` and `git clean -fd` to rollback ALL commits and remove untracked files. Execute `on_failure.cleanup` if defined. Log `ERROR [<task-id>] [<category>] <message>`. Set status=`failed` (Task Selection Algorithm pass 2 handles retries when attempts < max_attempts) +5. **Track**: Increment per-session task counter. If `max_tasks_per_session` reached, log STATS and STOP. +6. **Continue**: Immediately pick next task (zero idle time) + +### Stopping Conditions + +- All tasks `completed` +- All remaining tasks `failed` at max_attempts or blocked by failed dependencies +- `session_config.max_tasks_per_session` reached for this session +- `session_config.max_sessions` reached across all sessions +- User interrupts + +## Context Window Recovery Protocol + +When a new session starts and finds a task with `status: "in_progress"`: + +1. **Check git state**: + ```bash + git diff --stat # Uncommitted changes? + git log --oneline -5 # Recent commits since task started? + git stash list # Any stashed work? + ``` +2. **Check checkpoints**: Read the task's `checkpoints` array to determine last completed step +3. **Decision matrix** (verify recent commits belong to this task by checking commit messages for the task-id): + +| Uncommitted? | Recent task commits? | Checkpoints? | Action | +|---|---|---|---| +| No | No | None | Mark `failed` with `[SESSION_TIMEOUT] No progress detected`, increment attempts | +| No | No | Some | Verify file state matches checkpoint claims. If files reflect checkpoint progress, resume from last step. If not, mark `failed` — work was lost | +| No | Yes | Any | Run `validation.command`. If passes → mark `completed`. If fails → `git reset --hard <started_at_commit>`, mark `failed` | +| Yes | No | Any | Run validation WITH uncommitted changes present. If passes → commit, mark `completed`. If fails → `git reset --hard <started_at_commit>` + `git clean -fd`, mark `failed` | +| Yes | Yes | Any | Commit uncommitted changes, run `validation.command`. If passes → mark `completed`. If fails → `git reset --hard <started_at_commit>` + `git clean -fd`, mark `failed` | + +4. **Log recovery**: `[timestamp] [SESSION-N] RECOVERY [task-id] action="<action taken>" reason="<reason>"` + +## Error Handling & Recovery Strategies + +Each error category has a default recovery strategy: + +| Category | Default Recovery | Agent Action | +|----------|-----------------|--------------| +| `ENV_SETUP` | Re-run init, then STOP if still failing | Run `harness-init.sh` again immediately. If fails twice, log and stop — environment is broken | +| `TASK_EXEC` | Rollback via `git reset --hard <started_at_commit>`, retry | Verify `started_at_commit` exists (`git cat-file -t <hash>`). If missing, mark failed at max_attempts. Otherwise reset, run `on_failure.cleanup` if defined, retry if attempts < max_attempts | +| `TEST_FAIL` | Rollback via `git reset --hard <started_at_commit>`, retry | Reset to `started_at_commit`, analyze test output to identify fix, retry with targeted changes | +| `TIMEOUT` | Kill process, execute cleanup, retry | Wrap validation with `timeout <seconds> <command>`. On timeout, run `on_failure.cleanup`, retry (consider splitting task if repeated) | +| `DEPENDENCY` | Skip task, mark blocked | Log which dependency failed, mark task as `failed` with dependency reason | +| `SESSION_TIMEOUT` | Use Context Window Recovery Protocol | New session assesses partial progress via Recovery Protocol — may result in completion or failure depending on validation | + +**JSON corruption**: If `harness-tasks.json` cannot be parsed, check for `harness-tasks.json.bak` (written before each modification). If backup exists and is valid, restore from it. If no valid backup, log `ERROR [ENV_SETUP] harness-tasks.json corrupted and unrecoverable` and STOP — task metadata (validation commands, dependencies, cleanup) cannot be reconstructed from logs alone. + +**Backup protocol**: Before every write to `harness-tasks.json`, copy the current file to `harness-tasks.json.bak`. + +## Environment Initialization + +If `harness-init.sh` exists in the project root, run it at every session start. The script must be idempotent. + +Example `harness-init.sh`: +```bash +#!/bin/bash +set -e +npm install 2>/dev/null || pip install -r requirements.txt 2>/dev/null || true +curl -sf http://localhost:5432 >/dev/null 2>&1 || echo "WARN: DB not reachable" +npm test -- --bail --silent 2>/dev/null || echo "WARN: Smoke test failed" +echo "Environment health check complete" +``` + +## Standardized Log Format + +All log entries use grep-friendly format on a single line: + +``` +[ISO-timestamp] [SESSION-N] <TYPE> [task-id]? [category]? message +``` + +`[task-id]` and `[category]` are included when applicable (task-scoped entries). Session-level entries (`INIT`, `LOCK`, `STATS`) omit them. + +Types: `INIT`, `Starting`, `Completed`, `ERROR`, `CHECKPOINT`, `ROLLBACK`, `RECOVERY`, `STATS`, `LOCK`, `WARN` + +Error categories: `ENV_SETUP`, `TASK_EXEC`, `TEST_FAIL`, `TIMEOUT`, `DEPENDENCY`, `SESSION_TIMEOUT` + +Filtering: +```bash +grep "ERROR" harness-progress.txt # All errors +grep "ERROR" harness-progress.txt | grep "TASK_EXEC" # Execution errors only +grep "SESSION-3" harness-progress.txt # All session 3 activity +grep "STATS" harness-progress.txt # All session summaries +grep "CHECKPOINT" harness-progress.txt # All checkpoints +grep "RECOVERY" harness-progress.txt # All recovery actions +``` + +## Session Statistics + +At session end, update `harness-tasks.json`: increment `session_count`, set `last_session` to current timestamp. Then append: + +``` +[timestamp] [SESSION-N] STATS tasks_total=10 completed=7 failed=1 pending=2 blocked=0 attempts_total=12 checkpoints=23 +``` + +`blocked` is computed at stats time: count of pending tasks whose `depends_on` includes a permanently failed task. It is not a stored status value. + +## Init Command (`/harness init`) + +1. Create `harness-progress.txt` with initialization entry +2. Create `harness-tasks.json` with empty task list and default `session_config` +3. Optionally create `harness-init.sh` template (chmod +x) +4. Ask user: add harness files to `.gitignore`? + +## Status Command (`/harness status`) + +Read `harness-tasks.json` and `harness-progress.txt`, then display: + +1. Task summary: count by status (completed, failed, pending, blocked). `blocked` = pending tasks whose `depends_on` includes a permanently failed task (computed, not a stored status). +2. Per-task one-liner: `[status] task-id: title (attempts/max_attempts)` +3. Last 5 lines from `harness-progress.txt` +4. Session count and last session timestamp + +Does NOT acquire the lock (read-only operation). + +## Add Command (`/harness add`) + +Append a new task to `harness-tasks.json` with auto-incremented id (`task-NNN`), status `pending`, default `max_attempts: 3`, empty `depends_on`, and no validation command. Prompt user for optional fields: `priority`, `depends_on`, `validation.command`, `timeout_seconds`. Requires lock acquisition (modifies JSON). + +## Tool Dependencies + +Requires: Bash, file read/write, git. All harness operations must be executed from the project root directory. +Does NOT require: specific MCP servers, programming languages, or test frameworks.