Codeman/docs/archive/respawn-improvement-plan.md

Respawn Controller Idle Detection Improvement Plan

Executive Summary

The current respawn controller relies primarily on parsing terminal output to detect idle states. This approach is fragile and leads to false positives/negatives (e.g., the w3-reddit-analyse session).

Key insight: Claude Code provides direct, authoritative signals via hooks and files that definitively indicate session state. We're receiving some of these signals but not using them for idle detection!

---

Current Detection Layers (What We Have)

Layer	Signal	Source	Reliability
1	Completion message ("Worked for Xm Xs")	Terminal parsing	Medium - can miss edge cases
2	Output silence (configurable duration)	Terminal activity	Low - Claude can be processing silently
3	Token stability	Terminal parsing	Low - tokens don't change during I/O waits
4	Working pattern absence	Terminal parsing	Medium - patterns can be missed
5	AI idle check	Spawned Claude CLI	High but slow (90s timeout)

Problem: All layers depend on parsing terminal output, which is inherently unreliable.

---

Available Claude Code Signals (Not Fully Utilized)

1. Stop Hook ⭐ CRITICAL - DEFINITIVE SIGNAL

What it is: Fires when the main Claude Code agent finishes responding.

From docs: "Runs when the main Claude Code agent has finished responding. Does not run if the stoppage occurred due to a user interrupt."

Current status: We receive it via /api/hook-event but don't use it for idle detection!

Input received:

{
  "session_id": "abc123",
  "transcript_path": "~/.claude/projects/.../00893aaf.jsonl",
  "hook_event_name": "Stop",
  "stop_hook_active": true  // Important for preventing loops
}

Action needed: The Stop hook should be the PRIMARY idle detection signal. When Claude fires Stop, the agent has definitively finished its response cycle.

2. idle_prompt Notification ⭐ HIGH VALUE

What it is: Fires after 60+ seconds of idle time when Claude is waiting for user input.

From docs: "When Claude is waiting for user input (after 60+ seconds of idle time)"

Current status: We receive it but only forward it to the UI for notification display.

Action needed: Use idle_prompt as a definitive confirmation that Claude is idle. If we receive this, there's no need for AI idle checks or output silence timers.

3. Transcript JSONL File ⭐ HIGH VALUE

What it is: Complete conversation history at ~/.claude/projects/{project-hash}/{session-id}.jsonl

Current status: We already watch subagent transcripts but not the main session transcript.

Data available:

• Every message (user, assistant, system)
• Every tool call with inputs/outputs
• Progress events
• Structured, parseable JSON

Action needed:

• Monitor the main transcript file (path provided in every hook input)
• Parse the last few entries to detect:
  • Tool completion
  • Assistant message completion
  • Error states
  • Plan mode prompts

4. PostToolUse Hook - Tool Completion Tracking

What it is: Fires immediately after any tool completes successfully.

Use case: Track exactly when tools finish to understand execution flow.

Current status: Not implemented.

Action needed: Add PostToolUse hooks to track tool completion events.

5. SubagentStop Hook - Background Agent Completion

What it is: Fires when a subagent (Task tool) finishes responding.

Current status: Not implemented in hooks config (we watch JSONL files separately).

Action needed: Add to hooks config for redundant detection.

6. permission_prompt and elicitation_dialog - Blocking State Detection

What it is: Fires when Claude needs user input (permission or question).

Current status: We receive and use for auto-accept blocking.

Enhancement: Use as definitive "Claude is NOT idle - it's waiting for user action".

---

Proposed Architecture: Multi-Signal Idle Detection

New Detection Hierarchy

Priority 1 (Definitive):
  └── Stop hook received → CONFIRMED IDLE
  └── idle_prompt received → CONFIRMED IDLE (60s+ idle)

Priority 2 (Blocking):
  └── permission_prompt received → NOT IDLE (waiting for permission)
  └── elicitation_dialog received → NOT IDLE (waiting for answer)
  └── Working patterns in terminal → NOT IDLE

Priority 3 (Supporting):
  └── Transcript analysis → Check last entries for completion
  └── Output silence + token stability → Weak idle signal

Priority 4 (Fallback):
  └── AI idle check → Only if no definitive signals after timeout

State Machine Changes

                     ┌─────────────────────────────────────┐
                     │                                     │
                     ▼                                     │
    ┌─────────────────────┐                               │
    │      WATCHING       │◄──────────────────────────────┤
    └─────────────────────┘                               │
           │      │                                       │
           │      │ Stop hook or idle_prompt              │
           │      └────────────────────────┐              │
           │                               ▼              │
           │ Output silence          ┌────────────┐       │
           │ (no definitive signals) │ HOOK_IDLE  │───────┤
           │                         └────────────┘       │
           ▼                         (skip AI check)      │
    ┌────────────────────┐                               │
    │  CONFIRMING_IDLE   │                               │
    └────────────────────┘                               │
           │                                             │
           │ Silence confirmed                           │
           ▼                                             │
    ┌────────────────────┐                               │
    │   AI_CHECKING      │──── IDLE verdict ─────────────┤
    └────────────────────┘                               │
           │                                             │
           │ WORKING verdict                             │
           └─────────────────────────────────────────────┘

New State: hook_idle

When a definitive hook signal is received:

1. Skip AI idle check entirely (saves time and API calls)
2. Short confirmation period (2-3s) to handle race conditions
3. Proceed directly to respawn sequence

---

Implementation Plan

Phase 1: Use Stop Hook for Idle Detection ✅ COMPLETED

Files modified:

• src/respawn-controller.ts
• src/web/server.ts
• test/respawn-controller.test.ts

Changes implemented:

1. Added stopHookReceived, stopHookTime, idlePromptReceived, idlePromptTime fields to DetectionStatus
2. Added hookConfirmTimer for short confirmation after hook signal (3s)
3. Added signalStopHook() method:
   • Sets stopHookReceived = true and timestamp
   • Cancels any running AI check (hook is definitive)
   • Starts 3s confirmation timer
   • If no new output during confirmation → triggers respawn cycle
4. Added signalIdlePrompt() method:
   • Sets idlePromptReceived = true and timestamp
   • Immediately confirms idle (skips confirmation timer - 60s+ already proven)
5. Added resetHookState() to clear hook flags on:
   • Controller start
   • Working patterns detected
   • Cycle completion
6. Updated server.ts /api/hook-event endpoint to call:
   • controller.signalStopHook() for stop events
   • controller.signalIdlePrompt() for idle_prompt events
7. Updated getDetectionStatus():
   • Returns hook signal states
   • Sets confidence to 100% when hook received
   • Updates statusText to show hook status

Tests added (9 new tests in RespawnController Hook-Based Idle Detection describe block):

• should expose signalStopHook method
• should expose signalIdlePrompt method
• should set stopHookReceived in detection status when Stop hook signaled
• should include hook status in statusText when Stop hook received
• should trigger respawn cycle after Stop hook confirmation
• should immediately confirm idle when idle_prompt signaled (skip confirmation)
• should cancel Stop hook confirmation if working patterns detected
• should ignore Stop hook when not in watching state
• should have 100% confidence when hook signal is received

Detection status update (implemented):

interface DetectionStatus {
  /** Layer 0: Stop hook received (highest priority - definitive signal) */
  stopHookReceived: boolean;
  stopHookTime: number | null;
  /** Layer 0: idle_prompt notification received (definitive signal) */
  idlePromptReceived: boolean;
  idlePromptTime: number | null;

  // Existing fields...
}

Phase 2: Use idle_prompt for Definitive Idle ✅ COMPLETED (in Phase 1)

Already implemented in Phase 1:

1. signalIdlePrompt() method sets idlePromptReceived = true
2. Immediately calls onIdleConfirmed() - skips all other detection
3. Server.ts calls controller.signalIdlePrompt() when idle_prompt event received
4. 60s+ of Claude waiting = definitive idle signal

Phase 3: Transcript File Monitoring ✅ COMPLETED

New file: src/transcript-watcher.ts

Functionality implemented:

1. Watch the session's transcript JSONL file using fs.watch()
2. Parse new entries as they're appended (incremental reading from last position)
3. Detect:
   • result entry → transcript:complete event (isComplete = true)
   • tool_use content block → transcript:tool_start event
   • tool_result content block → transcript:tool_end event
   • AskUserQuestion or ExitPlanMode tools → transcript:plan_mode event
   • Error conditions in result entries
4. Emit structured events consumed by respawn controller

Integration implemented:

• transcript_path added to allowed hook data fields in sanitizeHookData()
• transcriptWatchers Map added to WebServer for per-session watchers
• startTranscriptWatcher() creates watcher and wires up events:
  • transcript:complete → controller.signalTranscriptComplete()
  • transcript:plan_mode → controller.signalTranscriptPlanMode()
• stopTranscriptWatcher() cleans up on session cleanup
• Hook events with transcript_path automatically start watching

RespawnController methods added:

• signalTranscriptComplete() - Supporting signal that can accelerate idle detection
• signalTranscriptPlanMode() - Cancels auto-accept timer (like elicitation)

Tests added (13 tests in test/transcript-watcher.test.ts):

• Initialization tests
• File watching tests (existing file, non-existent file, stop, updatePath)
• Entry processing tests (user entry, result entry, tool execution, plan mode, errors)
• State management tests

Phase 4: Enhanced Hook Configuration

Update src/hooks-config.ts:

export function generateHooksConfig(): { hooks: Record<string, unknown[]> } {
  return {
    hooks: {
      Notification: [
        { matcher: 'idle_prompt', hooks: [...] },
        { matcher: 'permission_prompt', hooks: [...] },
        { matcher: 'elicitation_dialog', hooks: [...] },
      ],
      Stop: [{ hooks: [...] }],
      // NEW: Add these
      PostToolUse: [
        { matcher: '*', hooks: [...] }  // Track all tool completions
      ],
      SubagentStop: [{ hooks: [...] }],
      PreCompact: [
        { matcher: '*', hooks: [...] }  // Track compaction
      ],
    },
  };
}

Phase 5: Confidence Scoring Overhaul

Replace current confidence calculation with weighted signals:

function calculateConfidence(): number {
  let confidence = 0;

  // Definitive signals (100% confidence)
  if (this.stopHookReceived) confidence = 100;
  if (this.idlePromptReceived) confidence = 100;

  // Blocking signals (0% confidence)
  if (this.permissionPromptReceived) return 0;
  if (this.elicitationReceived) return 0;
  if (this.workingPatternRecent) return 0;

  // Supporting signals (build up to ~80%)
  if (confidence < 100) {
    if (this.outputSilent) confidence += 30;
    if (this.tokensStable) confidence += 20;
    if (this.transcriptShowsCompletion) confidence += 30;
  }

  return Math.min(100, confidence);
}

---

Expected Benefits

Metric	Current	After Implementation
False positive rate	~15-20%	<5%
Detection latency	10-90s (AI check)	3-5s (hook-based)
API calls for AI check	Every idle detection	Only when hooks unavailable
Reliability	Medium	High (definitive signals)

---

Testing Strategy

Unit Tests

1. Stop hook triggers immediate idle confirmation
2. idle_prompt skips all other detection
3. permission_prompt blocks idle detection
4. Transcript parsing correctly identifies completion
5. Fallback to AI check when no hooks received

Integration Tests

1. End-to-end with real Claude session
2. Hook event delivery and handling
3. Transcript file monitoring
4. Race condition handling

Scenarios to Test

1. Normal completion → Stop hook → respawn
2. Long-running task → idle_prompt → respawn
3. Permission needed → wait for user action
4. AskUserQuestion → wait for user answer
5. Plan mode → auto-accept → continue
6. Hooks disabled/unavailable → fallback to AI check

---

Migration Path

1. Implement Phase 1 - Stop hook detection (low risk, high value)
2. Deploy and monitor - Verify Stop hooks are reliable
3. Implement Phase 2 - idle_prompt (simple addition)
4. Implement Phase 3 - Transcript monitoring (more complex)
5. Implement Phase 4 - Enhanced hooks (optional, for completeness)
6. Implement Phase 5 - Refactor confidence scoring

---

Open Questions

1. Stop hook reliability: Does it fire 100% of the time? Edge cases?
2. Transcript file location: Always at the path in hook input?
3. Hook delivery latency: How quickly do hooks fire after state change?
4. Race conditions: What if Stop hook and new work happen simultaneously?

---

References

• Claude Code Hooks Documentation
• Agent SDK Documentation
• Current implementation: src/respawn-controller.ts
• Hooks config: src/hooks-config.ts
• Subagent watcher: src/subagent-watcher.ts