mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-09-30 12:39:42 +02:00
feat(tmux): report a dead pane's exit from the batched pane list
Codeman creates every tmux pane with `remain-on-exit on`. When the agent exits,
tmux keeps the pane, the tmux session, and the `tmux attach-session` process
Codeman records as the session's pid, so no PTY exit handler fires and nothing
writes the exit down. tmux itself knows: it marks the pane dead and reports the
exit status. This reads that.
`PANE_LIST_FORMAT` gains `#{pane_dead}`, `#{pane_dead_status}` and
`#{pane_dead_signal}`, and `startPaneExitWatcher()` refreshes a
muxName-to-observation map from ONE batched `tmux list-panes -a` per tick. Boot
reconciliation already ran that same call, so it now fills the map too and
recovery starts with a reading.
The watcher owns its own interval rather than riding `startStatsCollection()`,
which the issue suggested. That collector is armed when a browser opens the
Monitor panel and DISARMED when it closes it, and boot skips it entirely unless
recovery found a live session, so a session created on a freshly booted server
would publish nothing and one browser could turn detection off for every other.
Measured on an isolated instance: a dead pane with status 0 reported nothing
until `POST /api/mux-sessions/stats/start` was called by hand. It is still one
batched read per tick; only the timer changed.
Three rules keep a positive answer trustworthy. A session answers only when
tmux listed exactly one pane for it, because Codeman never splits a pane and a
session the user split by hand has none that speaks for the agent. A pane
answers only when `#{pane_dead}` said 1 or 0, because an empty field is a tmux
that did not answer. An absent status stays absent rather than becoming 0:
measured on tmux 3.2a, a SIGKILLed pane reports neither a status nor a signal,
and calling that a clean exit would be wrong in the direction that matters.
Two guards stop a slow read undoing a fast one. `EXEC_TIMEOUT_MS` is 5000 ms
against a 2000 ms interval, so a read can outlive two ticks: one already in
flight suppresses the next, and a generation counter that every
`clearPaneExit()` bumps discards a read that started before a respawn or a
kill. An observation also carries its pane pid, so a second command in the same
pane that exits the same way starts a new timestamp rather than inheriting the
first death's.
A non-empty read of `list-panes -a` is authoritative for the whole socket, so
sessions missing from it are pruned, which also bounds the map as tmux sessions
come and go outside `killSession()`. A failed or empty read retracts nothing.
The manager reports the raw pane reading and applies no session-shape scoping,
because the remote-reconnect watcher beside it needs exactly that raw reading.
`parsePaneList` becomes `parsePaneRows`, returning one row per pane instead of
a name-to-pid map; reconciliation builds its map from the rows. The parser's
existing cases carry over unchanged, including the launchd/systemd literal-tab
regression from PR #71.
Refs Ark0N/Codeman#446.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
9466acfc1a
commit
02dc46dcd7
@@ -24,6 +24,7 @@ import type {
|
||||
OmpConfig,
|
||||
SessionRemote,
|
||||
SessionDocker,
|
||||
PaneExit,
|
||||
} from './types.js';
|
||||
|
||||
/**
|
||||
@@ -56,6 +57,16 @@ export interface MuxSession {
|
||||
respawnConfig?: PersistedRespawnConfig;
|
||||
/** Whether Ralph / Todo tracking is enabled */
|
||||
ralphEnabled?: boolean;
|
||||
/**
|
||||
* This record was rebuilt from the tmux socket rather than from Codeman's own
|
||||
* bookkeeping, so everything on it but the name and the pid is a guess. Its
|
||||
* synthetic `restored-<fragment>` id cannot find the session's `state.json`
|
||||
* entry either, which means a remote or docker session rediscovered this way
|
||||
* arrives with no `remote`/`docker` metadata and looks local. Anything that
|
||||
* would be WRONG about such a session rather than merely vague must fail
|
||||
* closed on this flag.
|
||||
*/
|
||||
discovered?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -180,6 +191,7 @@ export interface PaneCaptureOptions {
|
||||
* - `sessionKilled` (data: { sessionId: string }) - Session terminated
|
||||
* - `sessionDied` (data: { sessionId: string }) - Session died unexpectedly
|
||||
* - `statsUpdated` (sessions: MuxSessionWithStats[]) - Stats refreshed
|
||||
* - `paneExitsUpdated` () - A pane read finished; ask `getPaneExit()` per session
|
||||
*/
|
||||
export interface TerminalMultiplexer extends EventEmitter {
|
||||
/** Which backend this instance uses */
|
||||
@@ -294,6 +306,24 @@ export interface TerminalMultiplexer extends EventEmitter {
|
||||
/** Check if the pane in a session is dead (command exited but remain-on-exit keeps it alive) */
|
||||
isPaneDead(muxName: string): boolean;
|
||||
|
||||
/**
|
||||
* What the last pane read saw of this session's agent, or `undefined` for
|
||||
* UNKNOWN (Ark0N/Codeman#446). Unlike `isPaneDead()` this costs nothing: it
|
||||
* reads a map the batched watcher fills, so it answers no fresher than that
|
||||
* watcher's interval and the three synchronous `isPaneDead()` callers still
|
||||
* need their own probe. See {@link PaneExit}.
|
||||
*/
|
||||
getPaneExit?(muxName: string): PaneExit | undefined;
|
||||
|
||||
/** Forget a session's exit observation, e.g. once its pane has been respawned. */
|
||||
clearPaneExit?(muxName: string): void;
|
||||
|
||||
/** Start polling every pane on the socket for an exited agent. */
|
||||
startPaneExitWatcher?(intervalMs?: number): void;
|
||||
|
||||
/** Stop the pane-exit watcher. */
|
||||
stopPaneExitWatcher?(): void;
|
||||
|
||||
/** Respawn a dead pane with a fresh command. Returns the new PID or null on failure. */
|
||||
respawnPane(options: RespawnPaneOptions): Promise<number | null>;
|
||||
|
||||
|
||||
+297
-15
@@ -58,6 +58,7 @@ import {
|
||||
type SessionRemote,
|
||||
type SessionDocker,
|
||||
type DockerCommandMode,
|
||||
type PaneExit,
|
||||
} from './types.js';
|
||||
import { getCli } from './config/cli-registry/registry.js';
|
||||
import { missingCliMessage, resolveCliBinDir } from './utils/cli-resolver.js';
|
||||
@@ -152,6 +153,9 @@ const GRACEFUL_SHUTDOWN_WAIT_MS = 100;
|
||||
/** Default stats collection interval (2 seconds) */
|
||||
const DEFAULT_STATS_INTERVAL_MS = 2000;
|
||||
|
||||
/** How often the pane-exit watcher re-reads every pane on the socket. */
|
||||
const DEFAULT_PANE_EXIT_INTERVAL_MS = 2000;
|
||||
|
||||
/** Default remote-reconnect watcher poll interval (5 seconds) — COD-108 */
|
||||
const DEFAULT_REMOTE_RECONNECT_INTERVAL_MS = 5000;
|
||||
|
||||
@@ -219,8 +223,18 @@ const DEFAULT_CODEMAN_TMUX_SOCKET = DEFAULT_TMUX_SOCKET;
|
||||
*/
|
||||
const PANE_LIST_SEP = '|';
|
||||
|
||||
/** Format string for `tmux list-panes -F`. Keep in sync with {@link parsePaneList}. */
|
||||
const PANE_LIST_FORMAT = `#{session_name}${PANE_LIST_SEP}#{pane_pid}`;
|
||||
/**
|
||||
* Format string for `tmux list-panes -F`. Keep in sync with {@link parsePaneRows}.
|
||||
*
|
||||
* The three `pane_dead*` fields carry the agent-exit signal of Ark0N/Codeman#446.
|
||||
* Appending them is backward compatible in both directions. A tmux that does not
|
||||
* know a variable substitutes the empty string rather than failing, which is how
|
||||
* tmux 3.2a answers `#{pane_dead_signal}` (added in 3.4), and the parser reads a
|
||||
* short row as "pid known, deadness unknown" rather than discarding it.
|
||||
*/
|
||||
const PANE_LIST_FORMAT =
|
||||
`#{session_name}${PANE_LIST_SEP}#{pane_pid}` +
|
||||
`${PANE_LIST_SEP}#{pane_dead}${PANE_LIST_SEP}#{pane_dead_status}${PANE_LIST_SEP}#{pane_dead_signal}`;
|
||||
|
||||
/**
|
||||
* 构建 pane 启动前的 nofile 修复命令。
|
||||
@@ -235,26 +249,114 @@ export function buildNofileLimitCommand(targetLimit = CLAUDE_CODE_NOFILE_LIMIT):
|
||||
return `ulimit -Sn ${safeLimit} 2>/dev/null || ulimit -n ${safeLimit} 2>/dev/null || true`;
|
||||
}
|
||||
|
||||
/** One pane of one tmux session, as {@link parsePaneRows} reads it off the wire. */
|
||||
export interface PaneRow {
|
||||
/** tmux session this pane belongs to. Repeats once per pane of a split session. */
|
||||
sessionName: string;
|
||||
/** `#{pane_pid}` — the process tmux started in the pane. */
|
||||
pid: number;
|
||||
/** `#{pane_dead}` — true for 1, false for 0, undefined when tmux said nothing. */
|
||||
dead?: boolean;
|
||||
/** `#{pane_dead_status}` — the exit code, absent when tmux reported none. */
|
||||
exitStatus?: number;
|
||||
/** `#{pane_dead_signal}` — the killing signal, absent before tmux 3.4 and when unsignalled. */
|
||||
exitSignal?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the output of `tmux list-panes -a -F '#{session_name}|#{pane_pid}'`
|
||||
* into a Map of session-name → pane pid. Exported for unit testing.
|
||||
* One pane-exit reading, with the pane pid that produced it.
|
||||
*
|
||||
* The pid never leaves this module. It is what distinguishes "the same dead
|
||||
* pane, seen again" from "a second command in the same pane that also exited
|
||||
* with the same status", so the `at` stamp can hold across the first and must
|
||||
* not across the second. {@link PaneExit} itself stays free of it: the pid on
|
||||
* the session record is the attach client's, and a second pid there would
|
||||
* invite exactly the confusion Ark0N/Codeman#446 is about.
|
||||
*/
|
||||
export interface PaneExitObservation {
|
||||
/** `#{pane_pid}` of the pane this reading came from. */
|
||||
panePid: number;
|
||||
/** What to publish on the session record. */
|
||||
exit: PaneExit;
|
||||
}
|
||||
|
||||
/** Read one optional numeric field; a blank or non-numeric value is "not reported". */
|
||||
function paneField(fields: string[], index: number): number | undefined {
|
||||
const raw = fields[index];
|
||||
if (raw === undefined || raw === '') return undefined;
|
||||
const value = parseInt(raw, 10);
|
||||
return Number.isNaN(value) ? undefined : value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the output of `tmux list-panes -a -F` under {@link PANE_LIST_FORMAT}
|
||||
* into one row per pane, in tmux's own order. Exported for unit testing.
|
||||
*
|
||||
* - Skips empty lines and lines without the separator.
|
||||
* - Skips entries with a non-numeric pid or empty name.
|
||||
* - Leaves every field after the pid undefined when it is blank or absent, so a
|
||||
* row from an older tmux still yields its pid.
|
||||
*/
|
||||
export function parsePaneList(output: string): Map<string, number> {
|
||||
const result = new Map<string, number>();
|
||||
export function parsePaneRows(output: string): PaneRow[] {
|
||||
const rows: PaneRow[] = [];
|
||||
for (const line of output.split('\n')) {
|
||||
if (!line) continue;
|
||||
const sep = line.indexOf(PANE_LIST_SEP);
|
||||
if (sep === -1) continue;
|
||||
const name = line.slice(0, sep);
|
||||
const pid = parseInt(line.slice(sep + 1), 10);
|
||||
if (name && !Number.isNaN(pid)) {
|
||||
result.set(name, pid);
|
||||
}
|
||||
if (!line.includes(PANE_LIST_SEP)) continue;
|
||||
const fields = line.split(PANE_LIST_SEP);
|
||||
const sessionName = fields[0];
|
||||
const pid = parseInt(fields[1] ?? '', 10);
|
||||
if (!sessionName || Number.isNaN(pid)) continue;
|
||||
const deadFlag = fields[2];
|
||||
rows.push({
|
||||
sessionName,
|
||||
pid,
|
||||
dead: deadFlag === '1' ? true : deadFlag === '0' ? false : undefined,
|
||||
exitStatus: paneField(fields, 3),
|
||||
exitSignal: paneField(fields, 4),
|
||||
});
|
||||
}
|
||||
return result;
|
||||
return rows;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide, from every pane tmux listed, which tmux sessions have an exited agent.
|
||||
* Exported for unit testing. Returns one entry per session with a known answer;
|
||||
* a session absent from the map is UNKNOWN, which must never render as alive.
|
||||
*
|
||||
* Two rules make a positive answer trustworthy:
|
||||
*
|
||||
* A session answers only when tmux listed EXACTLY ONE pane for it. Codeman
|
||||
* creates one pane per session and `isPaneDead()` reads one pane, so a session
|
||||
* the user has split by hand has no single "the agent" to report on, and
|
||||
* guessing which of its panes speaks for the session could report a live
|
||||
* session as exited.
|
||||
*
|
||||
* A pane answers only when `#{pane_dead}` said 1 or 0. An empty field is a tmux
|
||||
* that did not answer, not a live pane.
|
||||
*
|
||||
* `status` and `signal` stay absent when tmux did not report them. Measured on
|
||||
* tmux 3.2a, a SIGKILLed pane reports neither, so folding an absent status into
|
||||
* 0 would turn an unexplained death into a clean exit.
|
||||
*/
|
||||
export function derivePaneExits(rows: PaneRow[], now: number): Map<string, PaneExitObservation> {
|
||||
const panesPerSession = new Map<string, number>();
|
||||
for (const row of rows) {
|
||||
panesPerSession.set(row.sessionName, (panesPerSession.get(row.sessionName) ?? 0) + 1);
|
||||
}
|
||||
const exits = new Map<string, PaneExitObservation>();
|
||||
for (const row of rows) {
|
||||
if (panesPerSession.get(row.sessionName) !== 1) continue;
|
||||
if (row.dead !== true) continue;
|
||||
exits.set(row.sessionName, {
|
||||
panePid: row.pid,
|
||||
exit: {
|
||||
...(row.exitStatus !== undefined ? { status: row.exitStatus } : {}),
|
||||
...(row.exitSignal !== undefined ? { signal: row.exitSignal } : {}),
|
||||
at: now,
|
||||
},
|
||||
});
|
||||
}
|
||||
return exits;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1527,6 +1629,30 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
private mouseSyncInterval: NodeJS.Timeout | null = null;
|
||||
/** Track last-known pane count per session to avoid unnecessary tmux set-option calls */
|
||||
private lastPaneCount: Map<string, number> = new Map();
|
||||
/**
|
||||
* muxName → the exited agent the pane-exit watcher last observed
|
||||
* (Ark0N/Codeman#446). Absence is the UNKNOWN arm of the tri-state, so an
|
||||
* entry goes the moment tmux stops reporting the pane dead, and the map is
|
||||
* empty until the first read runs. The manager reports what tmux says and
|
||||
* nothing more: the scoping that hides this for remote and docker sessions
|
||||
* lives on `Session`, because the remote-reconnect watcher above needs the
|
||||
* raw pane reading.
|
||||
*/
|
||||
private paneExits: Map<string, PaneExitObservation> = new Map();
|
||||
/** The pane-exit watcher's own interval. Runs whether or not stats are on. */
|
||||
private paneExitInterval: NodeJS.Timeout | null = null;
|
||||
/**
|
||||
* True while a pane read is in flight. `EXEC_TIMEOUT_MS` is 5000 ms against a
|
||||
* poll interval of 2000 ms, so without this a slow read overlaps the next two
|
||||
* and the older one can resolve last and win.
|
||||
*/
|
||||
private paneExitReadInFlight = false;
|
||||
/**
|
||||
* Bumped by every deliberate {@link clearPaneExit}. A read that started before
|
||||
* a clear carries the older generation and is discarded rather than writing
|
||||
* the death back over the pane that has just replaced it.
|
||||
*/
|
||||
private paneExitGeneration = 0;
|
||||
|
||||
// ── COD-108 remote-reconnect watcher state ────────────────────────────────
|
||||
/** Periodic watcher that re-establishes dropped remote sessions. */
|
||||
@@ -2297,6 +2423,11 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
);
|
||||
// Wait for the respawned process to start
|
||||
await new Promise((resolve) => setTimeout(resolve, TMUX_CREATION_WAIT_MS));
|
||||
// The pane now runs a fresh command, so whatever the last read observed of
|
||||
// the old one is history. Clearing it here rather than waiting for the next
|
||||
// poll also invalidates any read already in flight, which would otherwise
|
||||
// write the old death back over the pane that just replaced it.
|
||||
this.clearPaneExit(muxName);
|
||||
const pid = this.getPanePid(muxName);
|
||||
if (pid) session.pid = pid;
|
||||
return pid;
|
||||
@@ -2508,6 +2639,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
}
|
||||
}
|
||||
this.lastPaneCount.delete(session.muxName);
|
||||
this.clearPaneExit(session.muxName);
|
||||
this.sessions.delete(sessionId);
|
||||
this.clearRemoteReconnectState(sessionId);
|
||||
this.saveSessions();
|
||||
@@ -2618,6 +2750,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
}
|
||||
|
||||
this.lastPaneCount.delete(session.muxName);
|
||||
this.clearPaneExit(session.muxName);
|
||||
this.sessions.delete(sessionId);
|
||||
this.clearRemoteReconnectState(sessionId);
|
||||
this.saveSessions();
|
||||
@@ -2671,7 +2804,14 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
encoding: 'utf-8',
|
||||
timeout: EXEC_TIMEOUT_MS,
|
||||
}).trim();
|
||||
active = parsePaneList(output);
|
||||
const rows = parsePaneRows(output);
|
||||
active = new Map(rows.map((row) => [row.sessionName, row.pid]));
|
||||
// The same read answers both questions, so recovery starts with a pane-exit
|
||||
// reading rather than waiting for the first stats tick — which may never
|
||||
// come, since the collector only starts when boot found a live session.
|
||||
if (rows.length > 0) {
|
||||
this.applyPaneExits(derivePaneExits(rows, Date.now()));
|
||||
}
|
||||
} catch (err) {
|
||||
console.error('[TmuxManager] Failed to list tmux panes:', err);
|
||||
active = new Map();
|
||||
@@ -2686,6 +2826,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
} else {
|
||||
dead.push(sessionId);
|
||||
this.sessions.delete(sessionId);
|
||||
this.clearPaneExit(session.muxName);
|
||||
this.clearRemoteReconnectState(sessionId);
|
||||
this.emit('sessionDied', { sessionId });
|
||||
}
|
||||
@@ -2722,6 +2863,13 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
mode: 'claude',
|
||||
attached: false,
|
||||
name: `Restored: ${sessionName}`,
|
||||
// Every field above except the name and the pid is a guess: this record
|
||||
// was rebuilt from the socket because Codeman's own bookkeeping did not
|
||||
// have it. The synthetic id also cannot find the session's state.json
|
||||
// entry, so a remote or docker session rediscovered this way arrives
|
||||
// looking local. Consumers that would be wrong about such a session
|
||||
// read this flag and fail closed — see `Session.paneExitApplies`.
|
||||
discovered: true,
|
||||
};
|
||||
this.sessions.set(sessionId, session);
|
||||
knownMuxNames.add(sessionName);
|
||||
@@ -2882,6 +3030,138 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* What the last pane read saw of this tmux session's agent. `undefined` is
|
||||
* the UNKNOWN answer and must never be rendered as "alive": it covers a pane
|
||||
* that is running, a tmux session that no longer exists, a probe that failed,
|
||||
* and every poll that has not run yet. See {@link PaneExit}.
|
||||
*/
|
||||
getPaneExit(muxName: string): PaneExit | undefined {
|
||||
return this.paneExits.get(muxName)?.exit;
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-read every pane on the socket and refresh {@link paneExits}. ONE batched
|
||||
* `tmux list-panes -a` answers for every session at once, which is why this
|
||||
* polls rather than probing per session.
|
||||
*
|
||||
* A failed or empty probe leaves the previous answers ALONE rather than
|
||||
* clearing them. An empty read is "tmux did not answer", and clearing on it
|
||||
* would turn a transient failure into a silent retraction of a death Codeman
|
||||
* had already observed. A NON-empty read is different: `list-panes -a` lists
|
||||
* every pane on the socket, so it is authoritative and {@link applyPaneExits}
|
||||
* prunes against it.
|
||||
*
|
||||
* Two guards keep a slow read from undoing a fast one. A read already in
|
||||
* flight suppresses the next poll, and a read that started before a
|
||||
* {@link clearPaneExit} is discarded when it lands.
|
||||
*/
|
||||
async refreshPaneExits(now: number = Date.now()): Promise<void> {
|
||||
if (IS_TEST_MODE) return;
|
||||
if (this.paneExitReadInFlight) return;
|
||||
|
||||
const generation = this.paneExitGeneration;
|
||||
this.paneExitReadInFlight = true;
|
||||
let rows: PaneRow[];
|
||||
try {
|
||||
// execAsync, not execSync: this runs on a 2000 ms timer, and a synchronous
|
||||
// exec freezes the port while the process stays alive (see the
|
||||
// event-loop-monitor note in CLAUDE.md). The three `isPaneDead()` callers
|
||||
// stay synchronous because each is answering one request right then.
|
||||
const { stdout } = await execAsync(`${this.tmux()} list-panes -a -F '${PANE_LIST_FORMAT}' 2>/dev/null || true`, {
|
||||
encoding: 'utf-8',
|
||||
timeout: EXEC_TIMEOUT_MS,
|
||||
});
|
||||
rows = parsePaneRows(stdout.trim());
|
||||
} catch (err) {
|
||||
console.error('[TmuxManager] Failed to read pane exit state:', err);
|
||||
return;
|
||||
} finally {
|
||||
this.paneExitReadInFlight = false;
|
||||
}
|
||||
if (rows.length === 0) return;
|
||||
// A pane was respawned or killed while this read was out, so what it saw is
|
||||
// already history. Dropping it is what stops a freshly respawned pane from
|
||||
// being republished as exited.
|
||||
if (generation !== this.paneExitGeneration) return;
|
||||
|
||||
this.applyPaneExits(derivePaneExits(rows, now));
|
||||
}
|
||||
|
||||
/**
|
||||
* Fold one authoritative observation into {@link paneExits}. Split out from
|
||||
* the tmux call so the merge rules are unit-testable.
|
||||
*
|
||||
* `observed` comes from a read of EVERY pane on the socket, so a session
|
||||
* missing from it has no exit to report and its entry goes. That is what
|
||||
* keeps the map from growing without bound as tmux sessions come and go
|
||||
* outside `killSession()`. Only the caller may decide a read is authoritative:
|
||||
* a failed or empty one never reaches here.
|
||||
*
|
||||
* An entry keeps the `at` of the FIRST read that saw that exit, so the stamp
|
||||
* says when the agent was found gone rather than when the last poll ran. A
|
||||
* changed status, a changed signal, or a different pane pid all start a new
|
||||
* observation — the pid is what catches a second command in the same pane
|
||||
* that happened to exit the same way.
|
||||
*/
|
||||
applyPaneExits(observed: Map<string, PaneExitObservation>): void {
|
||||
for (const muxName of [...this.paneExits.keys()]) {
|
||||
if (!observed.has(muxName)) this.paneExits.delete(muxName);
|
||||
}
|
||||
for (const [muxName, next] of observed) {
|
||||
const prev = this.paneExits.get(muxName);
|
||||
const sameExit =
|
||||
prev !== undefined &&
|
||||
prev.panePid === next.panePid &&
|
||||
prev.exit.status === next.exit.status &&
|
||||
prev.exit.signal === next.exit.signal;
|
||||
this.paneExits.set(muxName, sameExit ? prev : next);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Forget a session's exit observation, e.g. once its pane has been respawned.
|
||||
* Also invalidates any read already in flight, so the answer this retracts
|
||||
* cannot be written back a moment later.
|
||||
*/
|
||||
clearPaneExit(muxName: string): void {
|
||||
this.paneExits.delete(muxName);
|
||||
this.paneExitGeneration++;
|
||||
}
|
||||
|
||||
/**
|
||||
* Poll for exited agents, on the manager's own interval.
|
||||
*
|
||||
* Deliberately NOT part of `startStatsCollection()`. That collector is armed
|
||||
* when the browser opens the Monitor panel and DISARMED when it closes it
|
||||
* (`panels-ui.js`), and it is skipped at boot entirely when no session was
|
||||
* recovered — so riding it would leave a session created on a freshly booted
|
||||
* server reporting nothing at all, and would let one browser turn exit
|
||||
* detection off for every other. Started unconditionally, like the mouse-mode
|
||||
* sync and the remote-reconnect watcher below.
|
||||
*
|
||||
* The `paneExitsUpdated` event is internal to the server; nothing here adds an
|
||||
* SSE event, and the field reaches the browser on `session:updated`.
|
||||
*/
|
||||
startPaneExitWatcher(intervalMs: number = DEFAULT_PANE_EXIT_INTERVAL_MS): void {
|
||||
if (this.paneExitInterval) {
|
||||
clearInterval(this.paneExitInterval);
|
||||
}
|
||||
this.paneExitInterval = setInterval(() => {
|
||||
if (IS_TEST_MODE) return;
|
||||
void this.refreshPaneExits()
|
||||
.then(() => this.emit('paneExitsUpdated'))
|
||||
.catch((err) => console.error('[TmuxManager] Pane exit watcher error:', err));
|
||||
}, intervalMs);
|
||||
}
|
||||
|
||||
stopPaneExitWatcher(): void {
|
||||
if (this.paneExitInterval) {
|
||||
clearInterval(this.paneExitInterval);
|
||||
this.paneExitInterval = null;
|
||||
}
|
||||
}
|
||||
|
||||
startStatsCollection(intervalMs: number = DEFAULT_STATS_INTERVAL_MS): void {
|
||||
if (this.statsInterval) {
|
||||
clearInterval(this.statsInterval);
|
||||
@@ -3101,6 +3381,8 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
|
||||
|
||||
destroy(): void {
|
||||
this.stopStatsCollection();
|
||||
this.stopPaneExitWatcher();
|
||||
this.paneExits.clear();
|
||||
this.stopMouseModeSync();
|
||||
this.stopRemoteReconnectWatcher();
|
||||
this.reconnectState.clear();
|
||||
|
||||
@@ -625,6 +625,40 @@ export interface CustomModelBookkeeping extends CustomModelSelection {
|
||||
launchModel?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* The agent inside a LOCAL tmux pane has exited, and the pane survived it.
|
||||
*
|
||||
* Codeman creates every pane with `remain-on-exit on`, so `/exit` ends the CLI
|
||||
* while tmux keeps the pane, the tmux session and the `tmux attach-session`
|
||||
* process Codeman records as the session's pid. No PTY exit handler runs, so
|
||||
* without this record the session reads as a live idle one (Ark0N/Codeman#446).
|
||||
*
|
||||
* The field is TRI-STATE, and the third state is the absence of the field:
|
||||
* `undefined` means Codeman does not know, and it must never be rendered as
|
||||
* "alive". It is absent for a direct-PTY session (no pane exists), for a remote
|
||||
* SSH session (the local pane holds the ssh client, whose death means transport
|
||||
* drop OR exit) and for a docker case (the local pane holds a `docker exec`
|
||||
* into the container's own tmux).
|
||||
*
|
||||
* `status` and `signal` are independently optional because tmux may know that
|
||||
* the pane died without reporting how. Measured on tmux 3.2a: a SIGKILLed pane
|
||||
* reports `pane_dead=1` with BOTH `#{pane_dead_status}` and `#{pane_dead_signal}`
|
||||
* empty, and `#{pane_dead_signal}` does not exist at all before tmux 3.4. So an
|
||||
* absent `status` means "the exit code is unknown", never "the exit code is 0".
|
||||
*/
|
||||
export interface PaneExit {
|
||||
/** tmux `#{pane_dead_status}` — the command's exit code. Absent when tmux reported none. */
|
||||
status?: number;
|
||||
/** tmux `#{pane_dead_signal}` — the signal that killed the command. Absent when unsignalled or unsupported. */
|
||||
signal?: number;
|
||||
/**
|
||||
* Wall-clock ms when THIS server process first observed the pane dead. It is
|
||||
* not when the agent exited, which nothing records, and a restart that finds
|
||||
* the pane still dead respawns it rather than re-timing the old exit.
|
||||
*/
|
||||
at: number;
|
||||
}
|
||||
|
||||
export interface SessionState {
|
||||
/** Unique session identifier */
|
||||
id: string;
|
||||
@@ -780,6 +814,21 @@ export interface SessionState {
|
||||
* (COD-118). Runtime-only: never restored on boot (fresh server = fresh breaker).
|
||||
*/
|
||||
respawnBlocked?: boolean;
|
||||
/**
|
||||
* The agent in this session's LOCAL tmux pane has exited (Ark0N/Codeman#446).
|
||||
* See {@link PaneExit} for the tri-state rule and for which session shapes
|
||||
* leave it absent. `status` and `pid` are deliberately untouched by it: the
|
||||
* PTY-exit breaker owns `status: 'error'`, and a null `pid` is what makes the
|
||||
* browser re-attach and launch a fresh CLI.
|
||||
*
|
||||
* Persisted so a reboot restore can tell a session whose agent exited from one
|
||||
* that was merely idle when the power went. `reboot-restore.ts` reads the
|
||||
* persisted record and never builds a `Session`, so the record is the only
|
||||
* place that survives the reboot to carry it. Nothing reads it there YET:
|
||||
* making the restore refuse such a session is a behavior change, and it
|
||||
* belongs with the part of Ark0N/Codeman#446 that closes exited sessions.
|
||||
*/
|
||||
paneExit?: PaneExit;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+170
-17
@@ -14,7 +14,8 @@ import {
|
||||
buildRemoteKillCommand,
|
||||
buildRemoteLaunchCommand,
|
||||
formatPaneSnapshot,
|
||||
parsePaneList,
|
||||
parsePaneRows,
|
||||
derivePaneExits,
|
||||
resolveActivePaneTarget,
|
||||
} from '../src/tmux-manager.js';
|
||||
import { execSync, exec } from 'node:child_process';
|
||||
@@ -910,41 +911,44 @@ describe('TmuxManager (unit)', () => {
|
||||
// exec without TTY). See PR #71.
|
||||
// ============================================================================
|
||||
|
||||
describe('parsePaneList', () => {
|
||||
describe('parsePaneRows', () => {
|
||||
/** Pull the name → pid map reconciliation builds, so these cases read as they used to. */
|
||||
const pids = (output: string) => new Map(parsePaneRows(output).map((row) => [row.sessionName, row.pid]));
|
||||
|
||||
it('parses well-formed output into name → pid', () => {
|
||||
const out = 'codeman-aaaa|1234\ncodeman-bbbb|5678\nclaudeman-cccc|9999';
|
||||
const result = parsePaneList(out);
|
||||
const result = pids(out);
|
||||
expect(result.size).toBe(3);
|
||||
expect(result.get('codeman-aaaa')).toBe(1234);
|
||||
expect(result.get('codeman-bbbb')).toBe(5678);
|
||||
expect(result.get('claudeman-cccc')).toBe(9999);
|
||||
});
|
||||
|
||||
it('returns an empty map for empty output', () => {
|
||||
expect(parsePaneList('').size).toBe(0);
|
||||
it('returns no rows for empty output', () => {
|
||||
expect(parsePaneRows('')).toEqual([]);
|
||||
});
|
||||
|
||||
it('skips blank lines', () => {
|
||||
const result = parsePaneList('\ncodeman-aaaa|100\n\n\ncodeman-bbbb|200\n');
|
||||
const result = pids('\ncodeman-aaaa|100\n\n\ncodeman-bbbb|200\n');
|
||||
expect(result.size).toBe(2);
|
||||
expect(result.get('codeman-aaaa')).toBe(100);
|
||||
expect(result.get('codeman-bbbb')).toBe(200);
|
||||
});
|
||||
|
||||
it('skips lines without the separator', () => {
|
||||
const result = parsePaneList('codeman-aaaa 1234\ncodeman-bbbb|5678');
|
||||
const result = pids('codeman-aaaa 1234\ncodeman-bbbb|5678');
|
||||
expect(result.size).toBe(1);
|
||||
expect(result.get('codeman-bbbb')).toBe(5678);
|
||||
});
|
||||
|
||||
it('skips lines with a non-numeric pid', () => {
|
||||
const result = parsePaneList('codeman-aaaa|notapid\ncodeman-bbbb|5678');
|
||||
const result = pids('codeman-aaaa|notapid\ncodeman-bbbb|5678');
|
||||
expect(result.size).toBe(1);
|
||||
expect(result.get('codeman-bbbb')).toBe(5678);
|
||||
});
|
||||
|
||||
it('skips lines with an empty session name', () => {
|
||||
const result = parsePaneList('|1234\ncodeman-bbbb|5678');
|
||||
const result = pids('|1234\ncodeman-bbbb|5678');
|
||||
expect(result.size).toBe(1);
|
||||
expect(result.get('codeman-bbbb')).toBe(5678);
|
||||
});
|
||||
@@ -955,15 +959,164 @@ describe('parsePaneList', () => {
|
||||
// tab byte. With the '|' separator, such literals must not be silently
|
||||
// treated as a delimiter — the line is discarded because there is no '|'.
|
||||
const literalBackslashT = 'codeman-aaaa\\t1234';
|
||||
const result = parsePaneList(literalBackslashT);
|
||||
expect(result.size).toBe(0);
|
||||
expect(parsePaneRows(literalBackslashT)).toEqual([]);
|
||||
});
|
||||
|
||||
it('splits on the first separator only', () => {
|
||||
// Numeric trailing junk after the pid is tolerated by parseInt — proves
|
||||
// that splitting on the first '|' leaves the pid extractable even if a
|
||||
// future tmux ever appended extra fields.
|
||||
const result = parsePaneList('codeman-aaaa|1234|extra-field');
|
||||
expect(result.get('codeman-aaaa')).toBe(1234);
|
||||
it('keeps a row whose pane_dead fields are missing, and calls its deadness unknown', () => {
|
||||
// A tmux old enough to have shipped the previous two-field format, or one
|
||||
// that dropped the trailing fields, must still yield its pid.
|
||||
const [row] = parsePaneRows('codeman-aaaa|1234');
|
||||
expect(row.pid).toBe(1234);
|
||||
expect(row.dead).toBeUndefined();
|
||||
expect(row.exitStatus).toBeUndefined();
|
||||
expect(row.exitSignal).toBeUndefined();
|
||||
});
|
||||
|
||||
it('reads a live pane as not dead, with no status or signal', () => {
|
||||
// Measured against tmux 3.2a: a live pane leaves both numeric fields blank.
|
||||
const [row] = parsePaneRows('codeman-aaaa|1234|0|||1');
|
||||
expect(row.dead).toBe(false);
|
||||
expect(row.exitStatus).toBeUndefined();
|
||||
expect(row.exitSignal).toBeUndefined();
|
||||
});
|
||||
|
||||
it('reads a dead pane with its exit status', () => {
|
||||
const [row] = parsePaneRows('codeman-aaaa|1234|1|7|');
|
||||
expect(row.dead).toBe(true);
|
||||
expect(row.exitStatus).toBe(7);
|
||||
expect(row.exitSignal).toBeUndefined();
|
||||
});
|
||||
|
||||
it('reads a dead pane with its killing signal', () => {
|
||||
const [row] = parsePaneRows('codeman-aaaa|1234|1||9');
|
||||
expect(row.dead).toBe(true);
|
||||
expect(row.exitStatus).toBeUndefined();
|
||||
expect(row.exitSignal).toBe(9);
|
||||
});
|
||||
|
||||
it('leaves a status of 0 as 0 rather than dropping it', () => {
|
||||
// The whole point of the field: a clean exit is the case part 2 acts on.
|
||||
const [row] = parsePaneRows('codeman-aaaa|1234|1|0|');
|
||||
expect(row.exitStatus).toBe(0);
|
||||
});
|
||||
|
||||
it('calls a non-numeric dead flag unknown rather than false', () => {
|
||||
const [row] = parsePaneRows('codeman-aaaa|1234|?||');
|
||||
expect(row.dead).toBeUndefined();
|
||||
});
|
||||
|
||||
it('returns one row per pane of a split session, in tmux order', () => {
|
||||
const rows = parsePaneRows('codeman-aaaa|100|0|||\ncodeman-aaaa|200|1|0|');
|
||||
expect(rows.map((row) => row.pid)).toEqual([100, 200]);
|
||||
expect(rows.map((row) => row.sessionName)).toEqual(['codeman-aaaa', 'codeman-aaaa']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('derivePaneExits', () => {
|
||||
const NOW = 1_700_000_000_000;
|
||||
|
||||
it('reports a single dead pane with its exit status', () => {
|
||||
const exits = derivePaneExits(parsePaneRows('codeman-aaaa|1234|1|0|'), NOW);
|
||||
expect(exits.get('codeman-aaaa')).toEqual({ panePid: 1234, exit: { status: 0, at: NOW } });
|
||||
});
|
||||
|
||||
it('reports a signalled death without inventing a status', () => {
|
||||
// Folding an absent status into 0 would turn an unexplained death into the
|
||||
// clean exit part 2 closes on sight.
|
||||
const exits = derivePaneExits(parsePaneRows('codeman-aaaa|1234|1||9'), NOW);
|
||||
expect(exits.get('codeman-aaaa')).toEqual({ panePid: 1234, exit: { signal: 9, at: NOW } });
|
||||
});
|
||||
|
||||
it('reports a death tmux could not explain at all', () => {
|
||||
// Measured on tmux 3.2a: a SIGKILLed pane reports pane_dead=1 and nothing else.
|
||||
const exits = derivePaneExits(parsePaneRows('codeman-aaaa|1234|1||'), NOW);
|
||||
expect(exits.get('codeman-aaaa')).toEqual({ panePid: 1234, exit: { at: NOW } });
|
||||
});
|
||||
|
||||
it('says nothing about a live pane', () => {
|
||||
const exits = derivePaneExits(parsePaneRows('codeman-aaaa|1234|0|||'), NOW);
|
||||
expect(exits.has('codeman-aaaa')).toBe(false);
|
||||
});
|
||||
|
||||
it('says nothing about a pane whose deadness tmux did not report', () => {
|
||||
expect(derivePaneExits(parsePaneRows('codeman-aaaa|1234'), NOW).size).toBe(0);
|
||||
});
|
||||
|
||||
it('says nothing about a session with more than one pane, even when all are dead', () => {
|
||||
// A session the user split by hand has no single "the agent" to report on,
|
||||
// and guessing which pane speaks for it could call a live session exited.
|
||||
const exits = derivePaneExits(parsePaneRows('codeman-aaaa|100|1|0|\ncodeman-aaaa|200|1|0|'), NOW);
|
||||
expect(exits.size).toBe(0);
|
||||
});
|
||||
|
||||
it("answers per session, so one session's split does not silence another", () => {
|
||||
const exits = derivePaneExits(
|
||||
parsePaneRows('codeman-aaaa|100|1|0|\ncodeman-bbbb|200|1|0|\ncodeman-bbbb|201|0|||'),
|
||||
NOW
|
||||
);
|
||||
expect([...exits.keys()]).toEqual(['codeman-aaaa']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('TmuxManager pane-exit bookkeeping', () => {
|
||||
const NOW = 1_700_000_000_000;
|
||||
|
||||
it('reports nothing before any tick has run', () => {
|
||||
const manager = new TmuxManager();
|
||||
expect(manager.getPaneExit('codeman-aaaa')).toBeUndefined();
|
||||
});
|
||||
|
||||
it('keeps the timestamp of the FIRST tick that saw an unchanged exit', () => {
|
||||
// The stamp says when the agent was found gone, so a pane that stays dead
|
||||
// must not have its age reset every two seconds.
|
||||
const manager = new TmuxManager();
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|100|1|0|'), NOW));
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|100|1|0|'), NOW + 2000));
|
||||
expect(manager.getPaneExit('codeman-aaaa')).toEqual({ status: 0, at: NOW });
|
||||
});
|
||||
|
||||
it('starts a new observation when the exit status changes', () => {
|
||||
const manager = new TmuxManager();
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|100|1|0|'), NOW));
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|100|1|137|'), NOW + 2000));
|
||||
expect(manager.getPaneExit('codeman-aaaa')).toEqual({ status: 137, at: NOW + 2000 });
|
||||
});
|
||||
|
||||
it('forgets the exit once the same session reports a live pane', () => {
|
||||
const manager = new TmuxManager();
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|100|1|0|'), NOW));
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|101|0|||'), NOW + 2000));
|
||||
expect(manager.getPaneExit('codeman-aaaa')).toBeUndefined();
|
||||
});
|
||||
|
||||
it('prunes an exit for a session an authoritative read did not mention', () => {
|
||||
// `list-panes -a` lists every pane on the socket, so a session missing from
|
||||
// a successful read has no pane at all and no exit to report. Keeping the
|
||||
// entry would grow the map forever as tmux sessions come and go outside
|
||||
// killSession(). A FAILED or empty read never reaches here — refreshPaneExits
|
||||
// returns before calling this, which is the case the next test covers.
|
||||
const manager = new TmuxManager();
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|100|1|0|'), NOW));
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-bbbb|200|1|0|'), NOW));
|
||||
expect(manager.getPaneExit('codeman-aaaa')).toBeUndefined();
|
||||
expect(manager.getPaneExit('codeman-bbbb')).toEqual({ status: 0, at: NOW });
|
||||
});
|
||||
|
||||
it('starts a new observation when the same status comes from a different pane pid', () => {
|
||||
// A second command in the same pane that also exited 0 is a NEW death, and
|
||||
// its `at` must say so. Only reachable when the respawn bypassed
|
||||
// respawnPane() — a hand-run `tmux respawn-pane` — since every Codeman path
|
||||
// clears the entry outright.
|
||||
const manager = new TmuxManager();
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|100|1|0|'), NOW));
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|101|1|0|'), NOW + 60_000));
|
||||
expect(manager.getPaneExit('codeman-aaaa')).toEqual({ status: 0, at: NOW + 60_000 });
|
||||
});
|
||||
|
||||
it('forgets an exit on request, which is what a respawned pane needs', () => {
|
||||
const manager = new TmuxManager();
|
||||
manager.applyPaneExits(derivePaneExits(parsePaneRows('codeman-aaaa|100|1|0|'), NOW));
|
||||
manager.clearPaneExit('codeman-aaaa');
|
||||
expect(manager.getPaneExit('codeman-aaaa')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user