feat(tmux): report a dead pane's exit from the batched pane list

Codeman creates every tmux pane with `remain-on-exit on`. When the agent exits,
tmux keeps the pane, the tmux session, and the `tmux attach-session` process
Codeman records as the session's pid, so no PTY exit handler fires and nothing
writes the exit down. tmux itself knows: it marks the pane dead and reports the
exit status. This reads that.

`PANE_LIST_FORMAT` gains `#{pane_dead}`, `#{pane_dead_status}` and
`#{pane_dead_signal}`, and `startPaneExitWatcher()` refreshes a
muxName-to-observation map from ONE batched `tmux list-panes -a` per tick. Boot
reconciliation already ran that same call, so it now fills the map too and
recovery starts with a reading.

The watcher owns its own interval rather than riding `startStatsCollection()`,
which the issue suggested. That collector is armed when a browser opens the
Monitor panel and DISARMED when it closes it, and boot skips it entirely unless
recovery found a live session, so a session created on a freshly booted server
would publish nothing and one browser could turn detection off for every other.
Measured on an isolated instance: a dead pane with status 0 reported nothing
until `POST /api/mux-sessions/stats/start` was called by hand. It is still one
batched read per tick; only the timer changed.

Three rules keep a positive answer trustworthy. A session answers only when
tmux listed exactly one pane for it, because Codeman never splits a pane and a
session the user split by hand has none that speaks for the agent. A pane
answers only when `#{pane_dead}` said 1 or 0, because an empty field is a tmux
that did not answer. An absent status stays absent rather than becoming 0:
measured on tmux 3.2a, a SIGKILLed pane reports neither a status nor a signal,
and calling that a clean exit would be wrong in the direction that matters.

Two guards stop a slow read undoing a fast one. `EXEC_TIMEOUT_MS` is 5000 ms
against a 2000 ms interval, so a read can outlive two ticks: one already in
flight suppresses the next, and a generation counter that every
`clearPaneExit()` bumps discards a read that started before a respawn or a
kill. An observation also carries its pane pid, so a second command in the same
pane that exits the same way starts a new timestamp rather than inheriting the
first death's.

A non-empty read of `list-panes -a` is authoritative for the whole socket, so
sessions missing from it are pruned, which also bounds the map as tmux sessions
come and go outside `killSession()`. A failed or empty read retracts nothing.

The manager reports the raw pane reading and applies no session-shape scoping,
because the remote-reconnect watcher beside it needs exactly that raw reading.

`parsePaneList` becomes `parsePaneRows`, returning one row per pane instead of
a name-to-pid map; reconciliation builds its map from the rows. The parser's
existing cases carry over unchanged, including the launchd/systemd literal-tab
regression from PR #71.

Refs Ark0N/Codeman#446.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/tmux-manager.ts

@@ -58,6 +58,7 @@ import {
   type SessionRemote,
   type SessionDocker,
   type DockerCommandMode,
+  type PaneExit,
 } from './types.js';
 import { getCli } from './config/cli-registry/registry.js';
 import { missingCliMessage, resolveCliBinDir } from './utils/cli-resolver.js';
@@ -152,6 +153,9 @@ const GRACEFUL_SHUTDOWN_WAIT_MS = 100;
 /** Default stats collection interval (2 seconds) */
 const DEFAULT_STATS_INTERVAL_MS = 2000;
 
+/** How often the pane-exit watcher re-reads every pane on the socket. */
+const DEFAULT_PANE_EXIT_INTERVAL_MS = 2000;
+
 /** Default remote-reconnect watcher poll interval (5 seconds) — COD-108 */
 const DEFAULT_REMOTE_RECONNECT_INTERVAL_MS = 5000;
 
@@ -219,8 +223,18 @@ const DEFAULT_CODEMAN_TMUX_SOCKET = DEFAULT_TMUX_SOCKET;
  */
 const PANE_LIST_SEP = '|';
 
-/** Format string for `tmux list-panes -F`. Keep in sync with {@link parsePaneList}. */
-const PANE_LIST_FORMAT = `#{session_name}${PANE_LIST_SEP}#{pane_pid}`;
+/**
+ * Format string for `tmux list-panes -F`. Keep in sync with {@link parsePaneRows}.
+ *
+ * The three `pane_dead*` fields carry the agent-exit signal of Ark0N/Codeman#446.
+ * Appending them is backward compatible in both directions. A tmux that does not
+ * know a variable substitutes the empty string rather than failing, which is how
+ * tmux 3.2a answers `#{pane_dead_signal}` (added in 3.4), and the parser reads a
+ * short row as "pid known, deadness unknown" rather than discarding it.
+ */
+const PANE_LIST_FORMAT =
+  `#{session_name}${PANE_LIST_SEP}#{pane_pid}` +
+  `${PANE_LIST_SEP}#{pane_dead}${PANE_LIST_SEP}#{pane_dead_status}${PANE_LIST_SEP}#{pane_dead_signal}`;
 
 /**
  * 构建 pane 启动前的 nofile 修复命令。
@@ -235,26 +249,114 @@ export function buildNofileLimitCommand(targetLimit = CLAUDE_CODE_NOFILE_LIMIT):
   return `ulimit -Sn ${safeLimit} 2>/dev/null || ulimit -n ${safeLimit} 2>/dev/null || true`;
 }
 
+/** One pane of one tmux session, as {@link parsePaneRows} reads it off the wire. */
+export interface PaneRow {
+  /** tmux session this pane belongs to. Repeats once per pane of a split session. */
+  sessionName: string;
+  /** `#{pane_pid}` — the process tmux started in the pane. */
+  pid: number;
+  /** `#{pane_dead}` — true for 1, false for 0, undefined when tmux said nothing. */
+  dead?: boolean;
+  /** `#{pane_dead_status}` — the exit code, absent when tmux reported none. */
+  exitStatus?: number;
+  /** `#{pane_dead_signal}` — the killing signal, absent before tmux 3.4 and when unsignalled. */
+  exitSignal?: number;
+}
+
 /**
- * Parse the output of `tmux list-panes -a -F '#{session_name}|#{pane_pid}'`
- * into a Map of session-name → pane pid. Exported for unit testing.
+ * One pane-exit reading, with the pane pid that produced it.
+ *
+ * The pid never leaves this module. It is what distinguishes "the same dead
+ * pane, seen again" from "a second command in the same pane that also exited
+ * with the same status", so the `at` stamp can hold across the first and must
+ * not across the second. {@link PaneExit} itself stays free of it: the pid on
+ * the session record is the attach client's, and a second pid there would
+ * invite exactly the confusion Ark0N/Codeman#446 is about.
+ */
+export interface PaneExitObservation {
+  /** `#{pane_pid}` of the pane this reading came from. */
+  panePid: number;
+  /** What to publish on the session record. */
+  exit: PaneExit;
+}
+
+/** Read one optional numeric field; a blank or non-numeric value is "not reported". */
+function paneField(fields: string[], index: number): number | undefined {
+  const raw = fields[index];
+  if (raw === undefined || raw === '') return undefined;
+  const value = parseInt(raw, 10);
+  return Number.isNaN(value) ? undefined : value;
+}
+
+/**
+ * Parse the output of `tmux list-panes -a -F` under {@link PANE_LIST_FORMAT}
+ * into one row per pane, in tmux's own order. Exported for unit testing.
  *
  * - Skips empty lines and lines without the separator.
  * - Skips entries with a non-numeric pid or empty name.
+ * - Leaves every field after the pid undefined when it is blank or absent, so a
+ *   row from an older tmux still yields its pid.
  */
-export function parsePaneList(output: string): Map<string, number> {
-  const result = new Map<string, number>();
+export function parsePaneRows(output: string): PaneRow[] {
+  const rows: PaneRow[] = [];
   for (const line of output.split('\n')) {
     if (!line) continue;
-    const sep = line.indexOf(PANE_LIST_SEP);
-    if (sep === -1) continue;
-    const name = line.slice(0, sep);
-    const pid = parseInt(line.slice(sep + 1), 10);
-    if (name && !Number.isNaN(pid)) {
-      result.set(name, pid);
-    }
+    if (!line.includes(PANE_LIST_SEP)) continue;
+    const fields = line.split(PANE_LIST_SEP);
+    const sessionName = fields[0];
+    const pid = parseInt(fields[1] ?? '', 10);
+    if (!sessionName || Number.isNaN(pid)) continue;
+    const deadFlag = fields[2];
+    rows.push({
+      sessionName,
+      pid,
+      dead: deadFlag === '1' ? true : deadFlag === '0' ? false : undefined,
+      exitStatus: paneField(fields, 3),
+      exitSignal: paneField(fields, 4),
+    });
   }
-  return result;
+  return rows;
+}
+
+/**
+ * Decide, from every pane tmux listed, which tmux sessions have an exited agent.
+ * Exported for unit testing. Returns one entry per session with a known answer;
+ * a session absent from the map is UNKNOWN, which must never render as alive.
+ *
+ * Two rules make a positive answer trustworthy:
+ *
+ * A session answers only when tmux listed EXACTLY ONE pane for it. Codeman
+ * creates one pane per session and `isPaneDead()` reads one pane, so a session
+ * the user has split by hand has no single "the agent" to report on, and
+ * guessing which of its panes speaks for the session could report a live
+ * session as exited.
+ *
+ * A pane answers only when `#{pane_dead}` said 1 or 0. An empty field is a tmux
+ * that did not answer, not a live pane.
+ *
+ * `status` and `signal` stay absent when tmux did not report them. Measured on
+ * tmux 3.2a, a SIGKILLed pane reports neither, so folding an absent status into
+ * 0 would turn an unexplained death into a clean exit.
+ */
+export function derivePaneExits(rows: PaneRow[], now: number): Map<string, PaneExitObservation> {
+  const panesPerSession = new Map<string, number>();
+  for (const row of rows) {
+    panesPerSession.set(row.sessionName, (panesPerSession.get(row.sessionName) ?? 0) + 1);
+  }
+  const exits = new Map<string, PaneExitObservation>();
+  for (const row of rows) {
+    if (panesPerSession.get(row.sessionName) !== 1) continue;
+    if (row.dead !== true) continue;
+    exits.set(row.sessionName, {
+      panePid: row.pid,
+      exit: {
+        ...(row.exitStatus !== undefined ? { status: row.exitStatus } : {}),
+        ...(row.exitSignal !== undefined ? { signal: row.exitSignal } : {}),
+        at: now,
+      },
+    });
+  }
+  return exits;
 }
 
 /**
@@ -1527,6 +1629,30 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
   private mouseSyncInterval: NodeJS.Timeout | null = null;
   /** Track last-known pane count per session to avoid unnecessary tmux set-option calls */
   private lastPaneCount: Map<string, number> = new Map();
+  /**
+   * muxName → the exited agent the pane-exit watcher last observed
+   * (Ark0N/Codeman#446). Absence is the UNKNOWN arm of the tri-state, so an
+   * entry goes the moment tmux stops reporting the pane dead, and the map is
+   * empty until the first read runs. The manager reports what tmux says and
+   * nothing more: the scoping that hides this for remote and docker sessions
+   * lives on `Session`, because the remote-reconnect watcher above needs the
+   * raw pane reading.
+   */
+  private paneExits: Map<string, PaneExitObservation> = new Map();
+  /** The pane-exit watcher's own interval. Runs whether or not stats are on. */
+  private paneExitInterval: NodeJS.Timeout | null = null;
+  /**
+   * True while a pane read is in flight. `EXEC_TIMEOUT_MS` is 5000 ms against a
+   * poll interval of 2000 ms, so without this a slow read overlaps the next two
+   * and the older one can resolve last and win.
+   */
+  private paneExitReadInFlight = false;
+  /**
+   * Bumped by every deliberate {@link clearPaneExit}. A read that started before
+   * a clear carries the older generation and is discarded rather than writing
+   * the death back over the pane that has just replaced it.
+   */
+  private paneExitGeneration = 0;
 
   // ── COD-108 remote-reconnect watcher state ────────────────────────────────
   /** Periodic watcher that re-establishes dropped remote sessions. */
@@ -2297,6 +2423,11 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
       );
       // Wait for the respawned process to start
       await new Promise((resolve) => setTimeout(resolve, TMUX_CREATION_WAIT_MS));
+      // The pane now runs a fresh command, so whatever the last read observed of
+      // the old one is history. Clearing it here rather than waiting for the next
+      // poll also invalidates any read already in flight, which would otherwise
+      // write the old death back over the pane that just replaced it.
+      this.clearPaneExit(muxName);
       const pid = this.getPanePid(muxName);
       if (pid) session.pid = pid;
       return pid;
@@ -2508,6 +2639,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
         }
       }
       this.lastPaneCount.delete(session.muxName);
+      this.clearPaneExit(session.muxName);
       this.sessions.delete(sessionId);
       this.clearRemoteReconnectState(sessionId);
       this.saveSessions();
@@ -2618,6 +2750,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
     }
 
     this.lastPaneCount.delete(session.muxName);
+    this.clearPaneExit(session.muxName);
     this.sessions.delete(sessionId);
     this.clearRemoteReconnectState(sessionId);
     this.saveSessions();
@@ -2671,7 +2804,14 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
         encoding: 'utf-8',
         timeout: EXEC_TIMEOUT_MS,
       }).trim();
-      active = parsePaneList(output);
+      const rows = parsePaneRows(output);
+      active = new Map(rows.map((row) => [row.sessionName, row.pid]));
+      // The same read answers both questions, so recovery starts with a pane-exit
+      // reading rather than waiting for the first stats tick — which may never
+      // come, since the collector only starts when boot found a live session.
+      if (rows.length > 0) {
+        this.applyPaneExits(derivePaneExits(rows, Date.now()));
+      }
     } catch (err) {
       console.error('[TmuxManager] Failed to list tmux panes:', err);
       active = new Map();
@@ -2686,6 +2826,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
       } else {
         dead.push(sessionId);
         this.sessions.delete(sessionId);
+        this.clearPaneExit(session.muxName);
         this.clearRemoteReconnectState(sessionId);
         this.emit('sessionDied', { sessionId });
       }
@@ -2722,6 +2863,13 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
         mode: 'claude',
         attached: false,
         name: `Restored: ${sessionName}`,
+        // Every field above except the name and the pid is a guess: this record
+        // was rebuilt from the socket because Codeman's own bookkeeping did not
+        // have it. The synthetic id also cannot find the session's state.json
+        // entry, so a remote or docker session rediscovered this way arrives
+        // looking local. Consumers that would be wrong about such a session
+        // read this flag and fail closed — see `Session.paneExitApplies`.
+        discovered: true,
       };
       this.sessions.set(sessionId, session);
       knownMuxNames.add(sessionName);
@@ -2882,6 +3030,138 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
     }));
   }
 
+  /**
+   * What the last pane read saw of this tmux session's agent. `undefined` is
+   * the UNKNOWN answer and must never be rendered as "alive": it covers a pane
+   * that is running, a tmux session that no longer exists, a probe that failed,
+   * and every poll that has not run yet. See {@link PaneExit}.
+   */
+  getPaneExit(muxName: string): PaneExit | undefined {
+    return this.paneExits.get(muxName)?.exit;
+  }
+
+  /**
+   * Re-read every pane on the socket and refresh {@link paneExits}. ONE batched
+   * `tmux list-panes -a` answers for every session at once, which is why this
+   * polls rather than probing per session.
+   *
+   * A failed or empty probe leaves the previous answers ALONE rather than
+   * clearing them. An empty read is "tmux did not answer", and clearing on it
+   * would turn a transient failure into a silent retraction of a death Codeman
+   * had already observed. A NON-empty read is different: `list-panes -a` lists
+   * every pane on the socket, so it is authoritative and {@link applyPaneExits}
+   * prunes against it.
+   *
+   * Two guards keep a slow read from undoing a fast one. A read already in
+   * flight suppresses the next poll, and a read that started before a
+   * {@link clearPaneExit} is discarded when it lands.
+   */
+  async refreshPaneExits(now: number = Date.now()): Promise<void> {
+    if (IS_TEST_MODE) return;
+    if (this.paneExitReadInFlight) return;
+
+    const generation = this.paneExitGeneration;
+    this.paneExitReadInFlight = true;
+    let rows: PaneRow[];
+    try {
+      // execAsync, not execSync: this runs on a 2000 ms timer, and a synchronous
+      // exec freezes the port while the process stays alive (see the
+      // event-loop-monitor note in CLAUDE.md). The three `isPaneDead()` callers
+      // stay synchronous because each is answering one request right then.
+      const { stdout } = await execAsync(`${this.tmux()} list-panes -a -F '${PANE_LIST_FORMAT}' 2>/dev/null || true`, {
+        encoding: 'utf-8',
+        timeout: EXEC_TIMEOUT_MS,
+      });
+      rows = parsePaneRows(stdout.trim());
+    } catch (err) {
+      console.error('[TmuxManager] Failed to read pane exit state:', err);
+      return;
+    } finally {
+      this.paneExitReadInFlight = false;
+    }
+    if (rows.length === 0) return;
+    // A pane was respawned or killed while this read was out, so what it saw is
+    // already history. Dropping it is what stops a freshly respawned pane from
+    // being republished as exited.
+    if (generation !== this.paneExitGeneration) return;
+
+    this.applyPaneExits(derivePaneExits(rows, now));
+  }
+
+  /**
+   * Fold one authoritative observation into {@link paneExits}. Split out from
+   * the tmux call so the merge rules are unit-testable.
+   *
+   * `observed` comes from a read of EVERY pane on the socket, so a session
+   * missing from it has no exit to report and its entry goes. That is what
+   * keeps the map from growing without bound as tmux sessions come and go
+   * outside `killSession()`. Only the caller may decide a read is authoritative:
+   * a failed or empty one never reaches here.
+   *
+   * An entry keeps the `at` of the FIRST read that saw that exit, so the stamp
+   * says when the agent was found gone rather than when the last poll ran. A
+   * changed status, a changed signal, or a different pane pid all start a new
+   * observation — the pid is what catches a second command in the same pane
+   * that happened to exit the same way.
+   */
+  applyPaneExits(observed: Map<string, PaneExitObservation>): void {
+    for (const muxName of [...this.paneExits.keys()]) {
+      if (!observed.has(muxName)) this.paneExits.delete(muxName);
+    }
+    for (const [muxName, next] of observed) {
+      const prev = this.paneExits.get(muxName);
+      const sameExit =
+        prev !== undefined &&
+        prev.panePid === next.panePid &&
+        prev.exit.status === next.exit.status &&
+        prev.exit.signal === next.exit.signal;
+      this.paneExits.set(muxName, sameExit ? prev : next);
+    }
+  }
+
+  /**
+   * Forget a session's exit observation, e.g. once its pane has been respawned.
+   * Also invalidates any read already in flight, so the answer this retracts
+   * cannot be written back a moment later.
+   */
+  clearPaneExit(muxName: string): void {
+    this.paneExits.delete(muxName);
+    this.paneExitGeneration++;
+  }
+
+  /**
+   * Poll for exited agents, on the manager's own interval.
+   *
+   * Deliberately NOT part of `startStatsCollection()`. That collector is armed
+   * when the browser opens the Monitor panel and DISARMED when it closes it
+   * (`panels-ui.js`), and it is skipped at boot entirely when no session was
+   * recovered — so riding it would leave a session created on a freshly booted
+   * server reporting nothing at all, and would let one browser turn exit
+   * detection off for every other. Started unconditionally, like the mouse-mode
+   * sync and the remote-reconnect watcher below.
+   *
+   * The `paneExitsUpdated` event is internal to the server; nothing here adds an
+   * SSE event, and the field reaches the browser on `session:updated`.
+   */
+  startPaneExitWatcher(intervalMs: number = DEFAULT_PANE_EXIT_INTERVAL_MS): void {
+    if (this.paneExitInterval) {
+      clearInterval(this.paneExitInterval);
+    }
+    this.paneExitInterval = setInterval(() => {
+      if (IS_TEST_MODE) return;
+      void this.refreshPaneExits()
+        .then(() => this.emit('paneExitsUpdated'))
+        .catch((err) => console.error('[TmuxManager] Pane exit watcher error:', err));
+    }, intervalMs);
+  }
+
+  stopPaneExitWatcher(): void {
+    if (this.paneExitInterval) {
+      clearInterval(this.paneExitInterval);
+      this.paneExitInterval = null;
+    }
+  }
+
   startStatsCollection(intervalMs: number = DEFAULT_STATS_INTERVAL_MS): void {
     if (this.statsInterval) {
       clearInterval(this.statsInterval);
@@ -3101,6 +3381,8 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer {
 
   destroy(): void {
     this.stopStatsCollection();
+    this.stopPaneExitWatcher();
+    this.paneExits.clear();
     this.stopMouseModeSync();
     this.stopRemoteReconnectWatcher();
     this.reconnectState.clear();