feat(terminal): renderer watchdog, atomic replay clear, fetch deadlines, reconnect recovery

Four ways the terminal can silently stop being correct — in each case the buffer keeps updating, nothing throws, and the only recourse is a reload. 1. Renderer freeze after backgrounding. iOS DISCARDS scheduled rAF callbacks when a PWA backgrounds, and xterm's RenderDebouncer only clears its `_animationFrame` handle from inside that callback — so one drop leaves it permanently set and every later refresh() early-returns. Parsing is decoupled from rendering, so bytes keep filling the buffer correctly while nothing paints. Codeman has exactly ONE xterm for the whole page load, so a single backgrounding wedges it until a reload. Adds a 2s liveness poll and `_kickRenderer()`, which does what the dropped `_innerRefresh` would have. 2. Replay clears raced live output. xterm's write() is async-queued while reset() is synchronous and, per upstream, "does not clear input buffers and does not reset the parser" — so bytes queued before a reset are parsed after it and fuse into the snapshot. Verified against the real xterm 6 here: write('p8'); reset(); write('rmissions') renders "p8rmissions". The main path was already safe via a queued erase; the needsRefresh and clearTerminal paths were not. All three now share one queued `\x1bc` (RIS), which unlike 3J/H/2J also resets modes, charsets, scroll regions and SGR state. 3. Output lost on WebSocket reconnect. Input frames carry seq+cid and are delivered exactly once; output frames carry nothing. ws.onopen re-sends dims and flushes queued input, and needsRefresh only fires on external-CLI startup and SSE backpressure drain — never on reconnect. Output produced while offline was simply absent afterwards. Interim fix: reaching onclose means the drop was unintentional, so the session is marked and the next open reconciles from the server buffer. Sequencing output is the follow-up. 4. Terminal captures had no deadline. No AbortController anywhere in the frontend, including `?full=1`, which the code itself calls "unbounded-ish work: at the default history limit it can be megabytes". Adds a budget that scales with full-vs-tail and with captures in flight, degrading to a plain fetch where AbortController is missing. Also: the service-worker precache was dead — the build content-hashes assets but sw.js listed pre-hash names, so 15 of 23 entries 404'd (verified against a running instance) and cache.add().catch() hid it. Offline still worked via runtime caching, but CACHE_NAME was a constant so activate's cleanup never deleted anything and every past release's assets accumulated. Both are now derived from the build manifest. Crash-trail entries are flattened and capped, since they are joined with \n into one value and one call site interpolates a server-controlled WS close reason. The watchdog reads xterm privates — there is no public API. Every access is optional-chained so a shape change degrades to a no-op. `_renderService` only exists after open(), which needs a real DOM, so the gate cannot assert the field path; test/xterm-private-api.test.ts pins the dependency range instead. Tests: 23 new (terminal-resilience, sw-precache-manifest, xterm-private-api), all pure/static so they run in the gate, which excludes the mobile suite. One static source guard in history-truncation-notice updated for the renamed call; the behaviour it pins is unchanged. Not verified: no browser available, so no runtime reproduction of the freeze and no real-device test of the reconnect path. Both warrant a device pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 14:09:42 +02:00 · 2026-09-22 12:24:52 +05:30
parent 9466acfc1a
commit c0422c4e21
10 changed files with 743 additions and 31 deletions
@@ -66,7 +66,18 @@ const _crashDiag = {
  // concurrent clients (desktop + phone) don't clobber each other.
  _pageId: Date.now().toString(36) + '-' + Math.random().toString(36).slice(2, 8),
  log(msg) {
-    const entry = `${new Date().toISOString().slice(11,23)} ${msg}`;
+    // Entries are joined with '\n' into ONE localStorage value and beaconed to
+    // the server, and some call sites interpolate text this client does not
+    // control (a WebSocket close `reason` arrives from the server). A newline
+    // in there forges extra entries in the trail; an unbounded string can fill
+    // the storage quota and silently kill every later breadcrumb. Flatten and
+    // cap. CodemanDiag is loaded before app.js, but guard anyway — a
+    // diagnostic that can throw is worse than no diagnostic.
+    const flat =
+      typeof CodemanDiag !== 'undefined' && CodemanDiag.sanitizeDiagEntry
+        ? CodemanDiag.sanitizeDiagEntry(msg)
+        : String(msg == null ? '' : msg).replace(/[\r\n\u2028\u2029]+/g, ' ').slice(0, 300);
+    const entry = `${new Date().toISOString().slice(11,23)} ${flat}`;
    this._entries.push(entry);
    if (this._entries.length > this._maxEntries) this._entries.shift();
    try { localStorage.setItem('codeman-crash-diag', this._entries.join('\n')); } catch {}
@@ -682,6 +693,10 @@ class CodemanApp {
    this._wsReady = false;      // True when WS is open and ready for I/O
    this._wsState = 'disconnected'; // connecting | connected | reconnecting | fallback | disconnected
    this._wsLastRecvAt = 0;     // ms timestamp of the last frame received on the active WS
+    // Session whose socket dropped unintentionally, so output produced during
+    // the outage is missing from its buffer. Output frames carry no sequence
+    // number, so the only recovery is to refetch on the next successful open.
+    this._wsOutputGapSession = null;

    // Terminal write batching with DEC 2026 sync support
    this.pendingWrites = [];
@@ -2555,6 +2570,61 @@ class CodemanApp {
    }
  }

+  /**
+   * Fetch a terminal capture under a deadline.
+   *
+   * Every terminal fetch used to run with no timeout at all, including
+   * `?full=1`, which _maybeRefetchFullHistory itself calls "unbounded-ish work:
+   * at the default history limit it can be megabytes". On a stalled mobile link
+   * that request hangs on the browser default with no retry, and the load-state
+   * machinery stays armed behind it.
+   *
+   * The budget scales with what is being asked for and with how many captures
+   * are already running (see CodemanFetchDeadline): a full scrollback on a slow
+   * uplink legitimately needs longer than a tail, and eight tabs resuming must
+   * not all expire together because each assumed it had the link to itself.
+   *
+   * An abort surfaces as a rejected fetch, which every caller already handles —
+   * they wrap these in try/catch and log. That is the point: a timeout becomes a
+   * recoverable error instead of an indefinite hang.
+   *
+   * @param {string} url
+   * @param {{full?: boolean}} [opts]
+   * @returns {Promise<Response>}
+   */
+  async _fetchTerminalCapture(url, opts = {}) {
+    const deadlineMs =
+      typeof CodemanFetchDeadline !== 'undefined'
+        ? CodemanFetchDeadline.terminalFetchDeadlineMs({
+            full: !!opts.full,
+            inflight: this._terminalCaptureInflight || 0,
+          })
+        : 45000;
+    // AbortSignal.timeout() is not on every browser Codeman supports, so drive
+    // it from a controller and always clear the timer — an uncancelled one
+    // would abort a LATER request that reused this controller's signal.
+    //
+    // Degrade to a plain fetch where AbortController is missing rather than
+    // throwing: a capture with no deadline is the behaviour every caller had
+    // before this helper existed, while a ReferenceError here would take out
+    // terminal replay entirely. The deadline is a safety net, not a dependency.
+    const canAbort = typeof AbortController === 'function';
+    const controller = canAbort ? new AbortController() : null;
+    const timer = controller ? setTimeout(() => controller.abort(), deadlineMs) : null;
+    this._terminalCaptureInflight = (this._terminalCaptureInflight || 0) + 1;
+    try {
+      return await (controller ? fetch(url, { signal: controller.signal }) : fetch(url));
+    } catch (err) {
+      if (err?.name === 'AbortError') {
+        _crashDiag.log(`TERMINAL FETCH TIMEOUT after ${deadlineMs}ms`);
+      }
+      throw err;
+    } finally {
+      if (timer !== null) clearTimeout(timer);
+      this._terminalCaptureInflight = Math.max(0, (this._terminalCaptureInflight || 1) - 1);
+    }
+  }
+
  async _onSessionNeedsRefresh(event = {}) {
    // Server sends this after SSE backpressure clears — terminal data was dropped,
    // so reload the buffer to recover from any display corruption.
@@ -2573,15 +2643,16 @@ class CodemanApp {
      // TUI modes still recover the whole picture, with the downgrade guard for
      // repaint-mode panes whose tmux capture can be smaller than xterm's buffer.
      const useFullHistory = this.sessions.get(sessionId)?.mode !== 'shell';
-      let res = await fetch(
+      let res = await this._fetchTerminalCapture(
        useFullHistory
          ? `/api/sessions/${sessionId}/terminal?full=1`
-          : `/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`
+          : `/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`,
+        { full: useFullHistory }
      );
      let headersReceivedAt = performance.now();
      let data = (await res.json())?.data ?? {};
      if (useFullHistory && data.terminalBuffer && this._replayWouldShrinkBuffer(data.terminalBuffer)) {
-        res = await fetch(`/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`);
+        res = await this._fetchTerminalCapture(`/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`);
        headersReceivedAt = performance.now();
        data = (await res.json())?.data ?? {};
      }
@@ -2596,8 +2667,11 @@ class CodemanApp {
        // meaningless across it — distance from the bottom is what survives.
        const before = this.terminal.buffer?.active;
        const linesFromBottom = before ? Math.max(0, (before.baseY || 0) - (before.viewportY || 0)) : 0;
-        this.terminal.clear();
-        this.terminal.reset();
+        // One queued clear, not clear()+reset(): both of those are synchronous
+        // and skip xterm's write queue, so live bytes still parsing would land
+        // after them and fuse into the buffer written below. See
+        // _resetTerminalForReplay.
+        this._resetTerminalForReplay();
        await this.chunkedTerminalWrite(
          data.terminalBuffer,
          TERMINAL_CHUNK_SIZE,
@@ -2642,12 +2716,13 @@ class CodemanApp {

      // Fetch buffer, clear terminal, write buffer, resize (no Ctrl+L needed)
      try {
-        const res = await fetch(`/api/sessions/${data.id}/terminal`);
+        const res = await this._fetchTerminalCapture(`/api/sessions/${data.id}/terminal`);
        const headersReceivedAt = performance.now();
        const termData = (await res.json())?.data ?? {};

-        this.terminal.clear();
-        this.terminal.reset();
+        // Queued clear — see _resetTerminalForReplay for why clear()+reset()
+        // cannot do this job.
+        this._resetTerminalForReplay();
        if (termData.terminalBuffer) {
          // Strip any DEC 2026 markers and write raw content
          // (markers don't help here - this is a static buffer reload, not live Ink redraws)
@@ -2972,6 +3047,18 @@ class CodemanApp {
        // Flush any durably-queued input over the fresh socket (covers frames a
        // prior half-open socket silently dropped, and input typed while offline).
        this._onWsReady(sessionId);
+        // Reconcile the output hole this drop left (see the ws.onclose note).
+        // Only after an unintentional close — a first connect has no gap, and
+        // refetching there would duplicate the buffer selectSession just wrote.
+        if (this._wsOutputGapSession === sessionId) {
+          this._wsOutputGapSession = null;
+          _crashDiag.log(`WS REOPEN: reconciling output gap for ${sessionId}`);
+          // Fire-and-forget: this is recovery, and a failure here must not stop
+          // the socket coming up. _onSessionNeedsRefresh already guards against
+          // running while a buffer load is in flight and against a tab switch
+          // landing this session's history in another session's terminal.
+          void this._onSessionNeedsRefresh({ id: sessionId });
+        }
      }
    };

@@ -3019,6 +3106,22 @@ class CodemanApp {
        `WS CLOSE code=${event.code} reason=${event.reason || ''} action=${plan.action} attempts=${this._wsReconnectAttempts || 0}`
      );

+      // Output frames carry no sequence number, so a socket that dropped left a
+      // hole in the terminal with nothing to replay it: ws.onopen re-sends dims
+      // and flushes queued INPUT, and `needsRefresh` only fires on external-CLI
+      // startup and on SSE backpressure drain — never here. Whatever the PTY
+      // produced while the link was down is simply absent from the buffer.
+      //
+      // Reaching onclose at all means the drop was NOT intentional
+      // (_disconnectWs nulls this handler first), so mark the gap and let the
+      // next successful open reconcile from the server's buffer.
+      //
+      // Scoped to the session that actually lost bytes: a user who switches
+      // sessions during an outage gets a clean intentional disconnect for the
+      // new one, and its freshly-loaded buffer must not be refetched because a
+      // DIFFERENT session's socket dropped.
+      this._wsOutputGapSession = sessionId;
+
      const stillActive = this.activeSessionId === sessionId;
      if (plan.action === 'give-up') {
        this._wsState = stillActive ? 'fallback' : 'disconnected';
@@ -5866,9 +5969,29 @@ class CodemanApp {
    }
  }

+  /**
+   * Clear the terminal for a replay, IN STREAM.
+   *
+   * xterm's `write()` is asynchronously queued (the WriteBuffer parses in ~12ms
+   * slices) while `Terminal.reset()` is synchronous and, by upstream's own
+   * documentation, "does not clear input buffers and does not reset the parser,
+   * thus the terminal will continue to apply pending input data". So bytes
+   * queued just before a `reset()` are parsed AFTER it and fuse into whatever
+   * snapshot is written next — measured upstream as `p8rmissions` rendered
+   * where `bypass permissions` belonged.
+   *
+   * A queued clear cannot race that way: it lands after the leftovers and
+   * before the snapshot, whatever the queue held. This function used to follow
+   * the sync `reset()` with a queued `\x1b[3J\x1b[H\x1b[2J`, which already got
+   * that right for CONTENT. RIS (`\x1bc`) additionally resets modes, charsets,
+   * scroll regions and SGR state, so leftover bytes cannot park the terminal in
+   * alt-screen or an odd scroll region and survive the clear.
+   *
+   * Callers may write the replacement content in as many chunks as they like —
+   * ordering within the queue is what matters, not writing it all at once.
+   */
  _resetTerminalForReplay() {
-    this.terminal.reset();
-    this.terminal.write('\x1b[3J\x1b[H\x1b[2J');
+    this.terminal.write('\x1bc');
  }

  _recordTerminalLoadTiming(timing) {
@@ -5932,7 +6055,7 @@ class CodemanApp {
    this._fullHistoryRepullInFlight = true;
    try {
      const requestStartedAt = performance.now();
-      const res = await fetch(`/api/sessions/${sessionId}/terminal?full=1`);
+      const res = await this._fetchTerminalCapture(`/api/sessions/${sessionId}/terminal?full=1`, { full: true });
      const headersReceivedAt = performance.now();
      const payload = (await res.json())?.data ?? {};
      const bodyParsedAt = performance.now();
@@ -6445,10 +6568,11 @@ class CodemanApp {
      const useFullHistory = session?.mode !== 'shell' && !this._fullHistoryLoaded.has(sessionId);
      if (useFullHistory) this._fullHistoryLoaded.add(sessionId);
      const fetchStartedAt = performance.now();
-      const res = await fetch(
+      const res = await this._fetchTerminalCapture(
        useFullHistory
          ? `/api/sessions/${sessionId}/terminal?full=1`
-          : `/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`
+          : `/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`,
+        { full: useFullHistory }
      );
      const headersReceivedAt = performance.now();
      if (this._isStaleSelect(selectGen)) {
@@ -1560,6 +1560,119 @@ function buildSplitPickerSessions(sessions, sessionOrder, excludeId, detachedIds
  return result;
 }

+// ── Renderer liveness ──────────────────────────────────────────────────────
+//
+// iOS DISCARDS scheduled requestAnimationFrame callbacks when a PWA goes to
+// the background — not deferred, never delivered. xterm's RenderDebouncer only
+// clears its `_animationFrame` handle from INSIDE that callback:
+//
+//   refresh() {
+//     if (this._animationFrame !== undefined) return;   // <- stale forever
+//     this._animationFrame = requestAnimationFrame(() => this._innerRefresh());
+//   }
+//   _innerRefresh() { this._animationFrame = undefined; ... }   // never runs
+//
+// So after one backgrounding the handle is permanently non-undefined and EVERY
+// later render request returns on line one. Parsing is decoupled from
+// rendering, so bytes keep filling the buffer correctly and nothing throws —
+// the terminal is simply frozen. Closing and reopening fixes it because that
+// constructs a new Terminal, and therefore a new debouncer.
+//
+// Codeman is MORE exposed than a per-session-terminal app: there is exactly one
+// xterm instance for the whole page load, so a single backgrounding can wedge
+// it until a full reload.
+//
+// This is the pure decision half. The signature that distinguishes this from
+// every other way a terminal can look stuck is that bytes were WRITTEN and the
+// element is VISIBLE, yet onRender has not fired since:
+//
+//   frozen   = wroteAt > renderedAt && now - wroteAt >= threshold && visible
+//
+// Deliberately NOT a "no output at all" check: a quiet terminal is the normal
+// state and must never be kicked. And `visible` is required because a hidden
+// terminal legitimately stops rendering (xterm pauses it), so kicking there
+// would fire constantly on every backgrounded tab.
+const RENDER_STALL_MS = 4000;
+
+// How often the watchdog checks. Deliberately coarse: the failure it catches is
+// permanent until healed, so detecting it a second late costs nothing, while a
+// tight interval would burn a wakeup per second on every idle phone.
+const RENDER_LIVENESS_POLL_MS = 2000;
+
+/**
+ * Should the renderer be kicked? Pure so the CI gate can cover it — the DOM
+ * half (cancelling the stale handle) lives in terminal-ui.js.
+ *
+ * @param {{wroteAt:number, renderedAt:number, now:number, visible:boolean,
+ *          thresholdMs?:number}} s
+ * @returns {boolean}
+ */
+function shouldKickRenderer(s) {
+  if (!s || !s.visible) return false;
+  const wroteAt = Number(s.wroteAt) || 0;
+  const renderedAt = Number(s.renderedAt) || 0;
+  const now = Number(s.now) || 0;
+  // Nothing written yet — a fresh terminal has no render to be missing.
+  if (wroteAt <= 0) return false;
+  // A render landed at or after the last write: the pipeline is alive.
+  if (renderedAt >= wroteAt) return false;
+  const threshold = Number.isFinite(s.thresholdMs) && s.thresholdMs > 0 ? s.thresholdMs : RENDER_STALL_MS;
+  return now - wroteAt >= threshold;
+}
+
+// ── Fetch deadlines ────────────────────────────────────────────────────────
+//
+// No terminal fetch carried any deadline, including `?full=1`, which the code
+// itself describes as "unbounded-ish work: at the default history limit it can
+// be megabytes". On a stalled mobile link that request hangs on the browser
+// default with no retry and no path back to a usable terminal short of a
+// reload.
+//
+// A single fixed timeout is wrong in both directions — too short for a full
+// scrollback capture on a slow uplink, too long for a small tail on a dead
+// connection. So the deadline is scaled by what is actually being asked for,
+// and by how many captures are already in flight: on a slow link those bytes
+// must drain before this request's own bytes start moving, and its timer is
+// already running the whole time.
+const FETCH_DEADLINE_TAIL_MS = 15000;
+const FETCH_DEADLINE_FULL_MS = 45000;
+const FETCH_DEADLINE_MAX_MS = 120000;
+
+/**
+ * Deadline in ms for a terminal capture.
+ *
+ * @param {{full?:boolean, inflight?:number}} s - `full` = the ?full=1 capture;
+ *   `inflight` = captures already running (this one included or not, it only
+ *   scales the budget).
+ * @returns {number}
+ */
+function terminalFetchDeadlineMs(s) {
+  const full = !!(s && s.full);
+  const base = full ? FETCH_DEADLINE_FULL_MS : FETCH_DEADLINE_TAIL_MS;
+  const inflight = Math.max(0, Number(s && s.inflight) || 0);
+  // Each already-queued capture gets the newcomer one more base budget to wait
+  // through. Linear rather than clever: the point is only that eight tabs
+  // resuming do not all time out together because each assumed it was alone.
+  return Math.min(FETCH_DEADLINE_MAX_MS, base * (1 + inflight));
+}
+
+// ── Diagnostics hygiene ────────────────────────────────────────────────────
+//
+// The crash trail is joined with '\n' into ONE localStorage value and beaconed
+// to the server, and at least one call site interpolates server-controlled text
+// (a WebSocket close `reason`). An embedded newline there forges extra entries
+// in the trail; an unbounded string can fill the storage quota. Both are cheap
+// to close, and the trail is something a user may be asked to paste into an
+// issue.
+const DIAG_ENTRY_MAX_CHARS = 300;
+
+/** Flatten a diagnostic message to one bounded, newline-free line. */
+function sanitizeDiagEntry(msg) {
+  return String(msg == null ? '' : msg)
+    .replace(/[\r\n\u2028\u2029]+/g, ' ')
+    .slice(0, DIAG_ENTRY_MAX_CHARS);
+}
+
 if (typeof window !== 'undefined') {
  window.CodemanHistoryFormat = { formatHistoryBytes, computeHistoryTruncationNotice, computeRewriteScrollLine };
  window.CodemanFilePaths = { absoluteFilePathPattern, previewsInFileViewer, FILE_PREVIEW_EXTENSIONS };
@@ -1569,4 +1682,12 @@ if (typeof window !== 'undefined') {
    buildSplitPickerSessions,
    SPLIT_PANE_MIN_WIDTH,
  };
+  window.CodemanRenderLiveness = { shouldKickRenderer, RENDER_STALL_MS, RENDER_LIVENESS_POLL_MS };
+  window.CodemanFetchDeadline = {
+    terminalFetchDeadlineMs,
+    FETCH_DEADLINE_TAIL_MS,
+    FETCH_DEADLINE_FULL_MS,
+    FETCH_DEADLINE_MAX_MS,
+  };
+  window.CodemanDiag = { sanitizeDiagEntry, DIAG_ENTRY_MAX_CHARS };
 }
@@ -18,7 +18,16 @@
 * @see src/push-store.ts -- server-side VAPID key management and subscription CRUD
 */

-const CACHE_NAME = 'codeman-v1';
+// Build identity. scripts/build.mjs rewrites this declaration after it content-
+// hashes the assets; the literal below is what dev serves, and dev wants a
+// stable key.
+//
+// Why the cache key MUST carry it: `activate` deletes every cache whose key is
+// not the current one, so the old constant key meant that cleanup never deleted
+// anything — hashed assets from every release ever deployed accumulated in one
+// bucket until the origin hit its storage quota.
+const BUILD_ID = 'dev';
+const CACHE_NAME = `codeman-${BUILD_ID}`;

 // Reverse-proxy base path: the worker is served at `<base>/sw.js`, so its own
 // location tells us the mount prefix ('' at root, or '/codeman'). Every URL below
@@ -27,27 +36,27 @@ const CACHE_NAME = 'codeman-v1';
 const SW_BASE = self.location.pathname.replace(/\/sw\.js$/, '');
 const B = (p) => (p && p[0] === '/' ? SW_BASE + p : p);

+// Content-hashed assets. scripts/build.mjs rewrites this declaration with the
+// filenames it actually emitted; dev has no hashing, so the empty literal below
+// is correct there and the unhashed modules are simply cached on first use by
+// the runtime handler further down.
+//
+// This list used to be maintained by hand with the PRE-hash names, which the
+// build then renamed — so in production every entry 404'd and the silent
+// `.catch()` in install swallowed all of it. Measured against a running
+// instance: 15 of 23 entries failed. Offline still worked, because the fetch
+// handler caches every successful GET at runtime, but the precache warmed
+// nothing while looking like it did. Deriving it from the same manifest that
+// renames the files is the only thing that keeps the two from drifting again.
+const HASHED_ASSETS = [];
+
 // Core app shell -- cached on install for instant startup
 const APP_SHELL = [
  '/',
-  '/styles.css',
-  '/mobile.css',
-  '/constants.js',
-  '/app.js',
-  '/api-client.js',
-  '/terminal-ui.js',
-  '/session-ui.js',
-  '/settings-ui.js',
-  '/panels-ui.js',
-  '/notification-manager.js',
-  '/mobile-handlers.js',
-  '/keyboard-accessory.js',
-  '/voice-input.js',
+  ...HASHED_ASSETS.map((p) => '/' + p),
  '/vendor/xterm.min.js',
  '/vendor/xterm-addon-fit.min.js',
  '/vendor/xterm-addon-unicode11.min.js',
-  '/vendor/xterm-zerolag-input.js',
-  '/vendor/xterm-predictive-echo.js',
  '/vendor/xterm.css',
  '/icon-192.png',
  '/icon-512.png',
@@ -543,6 +543,14 @@ Object.assign(CodemanApp.prototype, {
      this.terminal.onRender(() => this._syncMobileHelperTextareaToCursor());
    }

+    // Renderer liveness — see _startRenderLivenessWatchdog. Registered for every
+    // device, not just touch: the rAF-discard behaviour is worst on an iOS PWA
+    // but a stale handle wedges the debouncer identically anywhere it happens.
+    this.terminal.onRender(() => {
+      this._lastRenderAt = Date.now();
+    });
+    this._startRenderLivenessWatchdog();
+
    // CJK IME input — textarea in index.html, just wire up send
    this._cjkInput = null;
    if (typeof CjkInput !== 'undefined') {
@@ -3343,7 +3351,105 @@ Object.assign(CodemanApp.prototype, {
    return performance.now() - this._lastUserScrollUpAt < window.CodemanTerminalInput.USER_SCROLL_STICKY_SUPPRESS_MS;
  },

+  /**
+   * Watchdog for a frozen renderer.
+   *
+   * iOS DISCARDS scheduled requestAnimationFrame callbacks when a PWA goes to
+   * the background — not deferred, never delivered. xterm's RenderDebouncer
+   * only clears its `_animationFrame` handle from INSIDE that callback, so once
+   * one is dropped the handle stays permanently non-undefined and every later
+   * `refresh()` returns on its first line. Parsing is decoupled from rendering,
+   * so bytes keep filling the buffer correctly and nothing throws: the terminal
+   * is simply frozen until the page is reloaded.
+   *
+   * Codeman is more exposed than an app that mounts a terminal per session —
+   * there is exactly ONE xterm instance for the whole page load, so a single
+   * backgrounding can wedge it for the rest of the session.
+   *
+   * The heal is what `_innerRefresh` would have done: cancel the stale handle,
+   * clear the field, and request a full repaint (which schedules a fresh rAF).
+   * Cancelling a genuinely pending handle is harmless — the full repaint that
+   * follows covers whatever it was going to draw.
+   *
+   * Discipline for reaching into xterm privates, and it is not optional: every
+   * access is optional-chained and the whole body is wrapped, so a shape change
+   * upstream degrades to a no-op. A self-heal that can break the terminal it is
+   * healing is worse than no self-heal.
+   *
+   * ⚠️ The field path (`_core._renderService._renderDebouncer._animationFrame`)
+   * is validated against xterm 6.x and CANNOT be covered by the CI gate:
+   * `_renderService` is only constructed by `Terminal.open()`, which needs a
+   * real DOM, and the gate runs in node. `test/xterm-private-api.test.ts` pins
+   * the dependency RANGE instead, so a major bump fails there and sends someone
+   * to re-check this by hand; `test/terminal-resilience.test.ts` covers the
+   * decision half. If the path ever goes stale the watchdog silently stops
+   * healing — that is the failure mode to watch for, and why the range guard
+   * exists at all.
+   */
+  _startRenderLivenessWatchdog() {
+    this._stopRenderLivenessWatchdog();
+    this._lastRenderAt = Date.now();
+    this._lastTerminalWriteAt = 0;
+    this._renderLivenessTimer = setInterval(() => {
+      try {
+        if (typeof CodemanRenderLiveness === 'undefined') return;
+        const kick = CodemanRenderLiveness.shouldKickRenderer({
+          wroteAt: this._lastTerminalWriteAt || 0,
+          renderedAt: this._lastRenderAt || 0,
+          now: Date.now(),
+          // A hidden terminal legitimately stops rendering (xterm pauses it),
+          // so only a VISIBLE one that owes us a frame counts as frozen.
+          visible: document.visibilityState === 'visible' && !!this.terminal?.element?.isConnected,
+        });
+        if (!kick) return;
+        const kicked = this._kickRenderer();
+        _crashDiag.log(`RENDER STALL: kick=${kicked}`);
+        // Treat the kick as the render for accounting purposes either way, so a
+        // terminal we cannot heal logs once per stall rather than every tick.
+        this._lastRenderAt = Date.now();
+      } catch {
+        /* a watchdog must never throw into the interval */
+      }
+    }, RENDER_LIVENESS_POLL_MS);
+  },
+
+  _stopRenderLivenessWatchdog() {
+    if (this._renderLivenessTimer) {
+      clearInterval(this._renderLivenessTimer);
+      this._renderLivenessTimer = null;
+    }
+  },
+
+  /**
+   * Do what xterm's dropped `_innerRefresh` would have done. Never throws.
+   * @returns {boolean} true if a stale handle was found and cleared.
+   */
+  _kickRenderer() {
+    try {
+      const renderService = this.terminal?._core?._renderService;
+      const debouncer = renderService?._renderDebouncer;
+      if (!debouncer || typeof renderService.refreshRows !== 'function') return false;
+      const handle = debouncer._animationFrame;
+      if (handle === undefined) return false; // not wedged — nothing to clear
+      try {
+        cancelAnimationFrame(handle);
+      } catch {
+        /* a stale handle may no longer be cancellable; clearing it is the point */
+      }
+      debouncer._animationFrame = undefined;
+      renderService.refreshRows(0, Math.max(0, (this.terminal.rows || 1) - 1));
+      return true;
+    } catch {
+      return false;
+    }
+  },
+
  batchTerminalWrite(data) {
+    // Feed the renderer watchdog. Recorded before the buffer-load early return
+    // below: a write that is queued rather than written still means the pipeline
+    // owes us a frame once it drains.
+    this._lastTerminalWriteAt = Date.now();
+
    // If a buffer load (chunkedTerminalWrite) is in progress, queue live events
    // to prevent interleaving historical buffer data with live SSE data.
    // This is critical: interleaving causes cursor position chaos with Ink redraws.