Files
Codeman/src/remote-reconnect.ts
T
Aamer Akhter 6dba8b5227 COD-108 auto-reconnect remote tmux sessions on SSH drop
Continuous remote-only reconnect watcher closing the COD-104 durability
arc: when a remote session's local ssh pane dies mid-run, re-establish it
automatically instead of leaving a dead pane until the user pokes it.

Design decisions (per cod108 design doc):
- D1 event->owner: TmuxManager watcher DETECTS a dead remote pane and emits
  `remoteSessionDropped`; the session owner (server) reassembles the same
  RespawnPaneOptions and calls Session.reattachRemote() -> respawnPane, which
  re-runs the idempotent remote command (owned new-session -A / non-owned
  attach) and REJOINS the still-running durable remote tmux session. The
  watcher never reassembles options itself, and never routes through the
  Claude-idle respawn-controller.
- D2 bounded backoff: per-session exponential backoff [5s,15s,45s,2m,5m,5m],
  reset on a successful reattach, `remoteReconnectExhausted` emitted once after
  the cap. Pure, unit-tested schedule + eligibility decision.
- D3 always-on + kill-switch: `remoteAutoReconnect` app setting (default ON),
  read each tick; when false the watcher does nothing.

Guards: killSession() (incl. the non-owned DETACH early-return) and shutdown
add the session to an intentional-teardown guard set + clear its backoff
BEFORE teardown, so a closed/killed tab is never auto-revived. Exactly one
reconnect in flight per session (inFlight guard prevents stacked respawns).
Per-session reconnect/guard state cleared on session removal.

New: src/remote-reconnect.ts (pure backoff + decideReconnect), TmuxManager
startRemoteReconnectWatcher/stop + runRemoteReconnectTick + noteRemoteReconnect
+ guardRemoteReconnect + clearRemoteReconnectState; Session.reattachRemote()
(+ extracted _buildRespawnPaneOptions, shared with interactive start); server
wiring + watcher start; 3 SSE events (sse-events.ts + constants.js in sync,
broadcast + app.js exhausted "Reconnect" affordance); remoteAutoReconnect
schema + settings-ui toggle.

Tests: test/remote-auto-reconnect.test.ts (21) - pure schedule, eligibility
(guarded never reconnects, non-remote/pane-alive/not-due skip, over-cap
exhaust), and manager-level integration (dead remote pane -> dropped ->
backoff -> exhausted; guarded emits nothing; reset-on-success; kill-switch
off; state-cleared-on-remove). Verified real-remote against aa-desktop: drop
local ssh pane -> watcher emitted -> respawnPane reattached the SAME remote
session (remote pane_pid unchanged 3939->3939); test session cleaned up, the
real host sessions left untouched.

Checks: tsc, eslint, check:frontend-syntax, check:public-assets, prettier
--check, build all green; tmux-manager/session-routes/session-manager/
sse-registry-parity suites pass.

(cherry picked from commit d13d58b1994eb6594fd2eadea208104d36204f9d)
2026-07-17 15:59:36 -04:00

185 lines
7.5 KiB
TypeScript

/**
* @fileoverview Pure logic for the remote-session auto-reconnect watcher (COD-108).
*
* COD-104 made remote tmux sessions durable + idempotently reattachable, but a
* reconnect only fired at explicit trigger points. COD-108 adds a continuous
* watcher (in `TmuxManager`) that detects a dead remote pane and emits
* `remoteSessionDropped`; `SessionManager`/server then reassembles the respawn
* options and reattaches (re-running the idempotent remote command).
*
* This module holds the SIDE-EFFECT-FREE pieces so they can be unit-tested
* without real tmux:
* - the bounded exponential **backoff schedule** (attempt → delay, capped),
* - the per-session **reconnect state** shape,
* - the **eligibility decision** (`decideReconnect`) given a session + its
* reconnect state + the current time + the guard set.
*
* The watcher in `tmux-manager.ts` owns the live `isPaneDead` probe and the
* timers; everything here is pure and deterministic (time is injected).
*
* @module remote-reconnect
*/
/**
* Bounded exponential backoff delays (ms) between reconnect attempts.
* Attempt N (1-based) waits `BACKOFF_SCHEDULE_MS[N-1]` from the previous emit
* before the next emit is eligible. After the last entry the session is
* considered `reconnect-exhausted` and the watcher stops emitting for it.
*
* 5s, 15s, 45s, 2m, 5m, 5m → ~6 attempts spanning ~13 minutes.
*/
export const BACKOFF_SCHEDULE_MS: readonly number[] = [5_000, 15_000, 45_000, 120_000, 300_000, 300_000];
/** Maximum number of reconnect attempts before exhaustion. */
export const MAX_RECONNECT_ATTEMPTS = BACKOFF_SCHEDULE_MS.length;
/**
* Delay (ms) to wait AFTER emitting attempt `attempt` (1-based) before the next
* attempt is eligible. `attempt <= 0` returns the first delay; an attempt at or
* beyond the cap returns the last delay (callers should check exhaustion via
* {@link isExhausted} rather than relying on this for the stop decision).
*
* Pure — no clock, no I/O.
*/
export function reconnectDelayForAttempt(attempt: number): number {
if (!Number.isFinite(attempt) || attempt <= 1) return BACKOFF_SCHEDULE_MS[0];
const idx = Math.min(Math.floor(attempt) - 1, BACKOFF_SCHEDULE_MS.length - 1);
return BACKOFF_SCHEDULE_MS[idx];
}
/** Whether `attempts` reconnect emits have reached/exceeded the cap. Pure. */
export function isExhausted(attempts: number): boolean {
return attempts >= MAX_RECONNECT_ATTEMPTS;
}
/**
* Per-session reconnect bookkeeping held by the watcher. All time values are
* epoch ms. `inFlight` guards against stacking respawns when a tick fires while
* a previous reattach is still running. `exhaustedEmitted` ensures the
* `remoteReconnectExhausted` event fires at most once per session.
*/
export interface RemoteReconnectState {
/** Number of `remoteSessionDropped` emits so far (advances per emit). */
attempts: number;
/** Earliest time (epoch ms) the next emit is eligible. 0 = eligible now. */
nextEligibleAt: number;
/** A reattach triggered by a prior emit is currently running. */
inFlight: boolean;
/** Cap reached — stop auto-retrying for this session. */
exhausted: boolean;
/** The `remoteReconnectExhausted` SSE event has already been emitted. */
exhaustedEmitted: boolean;
}
/** A fresh reconnect state (no attempts, immediately eligible). Pure. */
export function freshReconnectState(): RemoteReconnectState {
return { attempts: 0, nextEligibleAt: 0, inFlight: false, exhausted: false, exhaustedEmitted: false };
}
/**
* Advance the backoff after an emit at time `now`. Increments `attempts` and
* schedules `nextEligibleAt = now + delay`. Returns a NEW state object (does
* not mutate the input). Pure.
*
* NOTE: this does NOT set `exhausted`. Exhaustion is a decision the watcher
* makes on the FOLLOWING tick (via {@link decideReconnect} → `exhaust`), so the
* `remoteReconnectExhausted` event fires exactly once after the final attempt's
* backoff window elapses — not pre-emptively on the last emit.
*/
export function advanceBackoff(state: RemoteReconnectState, now: number): RemoteReconnectState {
const attempts = state.attempts + 1;
const delay = reconnectDelayForAttempt(attempts);
return {
...state,
attempts,
nextEligibleAt: now + delay,
};
}
/** Reset after a successful reattach — back to a fresh, eligible state. Pure. */
export function resetReconnectState(): RemoteReconnectState {
return freshReconnectState();
}
/** Minimal session view the decision needs (avoids importing MuxSession here). */
export interface ReconnectSessionView {
sessionId: string;
/** Truthy when this is a remote (SSH-wrapped) session. */
isRemote: boolean;
/** Result of `isPaneDead(muxName)` for this session. */
paneDead: boolean;
}
/**
* Decision outcomes for a single watcher tick on one session.
* - `emit` → emit `remoteSessionDropped { sessionId, attempt }`, then
* advance backoff (attempt = the returned `attempt`).
* - `exhaust` → cap reached this tick; emit `remoteReconnectExhausted` once.
* - `skip` → do nothing (not remote / pane alive / guarded / in-flight /
* not yet due / already exhausted).
*/
export type ReconnectAction =
| { kind: 'emit'; attempt: number }
| { kind: 'exhaust' }
| { kind: 'skip'; reason: ReconnectSkipReason };
export type ReconnectSkipReason =
| 'not-remote'
| 'pane-alive'
| 'guarded'
| 'in-flight'
| 'not-due'
| 'exhausted'
| 'disabled';
export interface DecideReconnectInput {
session: ReconnectSessionView;
state: RemoteReconnectState | undefined;
/** Session is in the intentional-teardown guard set (killed/detached/stopping). */
guarded: boolean;
/** Kill-switch: `remoteAutoReconnect` setting. When false, never reconnect. */
enabled: boolean;
now: number;
}
/**
* PURE eligibility decision for one session on one tick. No clock, no I/O — all
* inputs are passed in. The watcher translates the result into emits + state
* transitions.
*
* Order of guards (most-decisive first):
* 1. kill-switch off → skip:disabled
* 2. not a remote session → skip:not-remote
* 3. pane is alive → skip:pane-alive
* 4. intentional teardown guard → skip:guarded (NEVER revive a killed tab)
* 5. a reattach already running → skip:in-flight (no stacked respawns)
* 6. already exhausted → skip:exhausted (one exhaust emit, then quiet)
* 7. cap reached this tick → exhaust
* 8. not yet due (backoff) → skip:not-due
* 9. otherwise → emit (attempt = attempts + 1)
*/
export function decideReconnect(input: DecideReconnectInput): ReconnectAction {
const { session, state, guarded, enabled, now } = input;
if (!enabled) return { kind: 'skip', reason: 'disabled' };
if (!session.isRemote) return { kind: 'skip', reason: 'not-remote' };
if (!session.paneDead) return { kind: 'skip', reason: 'pane-alive' };
// Intentional kill / detach must NEVER be auto-revived.
if (guarded) return { kind: 'skip', reason: 'guarded' };
const s = state ?? freshReconnectState();
// Only one reconnect in flight per session — don't stack respawns.
if (s.inFlight) return { kind: 'skip', reason: 'in-flight' };
if (s.exhausted) return { kind: 'skip', reason: 'exhausted' };
// Cap reached: surface exhaustion once, then go quiet.
if (isExhausted(s.attempts)) return { kind: 'exhaust' };
// Backoff gate — only emit when due.
if (now < s.nextEligibleAt) return { kind: 'skip', reason: 'not-due' };
return { kind: 'emit', attempt: s.attempts + 1 };
}