mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-10-11 01:39:41 +02:00
feat(terminal): renderer watchdog, atomic replay clear, fetch deadlines, reconnect recovery
Four ways the terminal can silently stop being correct — in each case the
buffer keeps updating, nothing throws, and the only recourse is a reload.
1. Renderer freeze after backgrounding. iOS DISCARDS scheduled rAF callbacks
when a PWA backgrounds, and xterm's RenderDebouncer only clears its
`_animationFrame` handle from inside that callback — so one drop leaves it
permanently set and every later refresh() early-returns. Parsing is
decoupled from rendering, so bytes keep filling the buffer correctly while
nothing paints. Codeman has exactly ONE xterm for the whole page load, so a
single backgrounding wedges it until a reload. Adds a 2s liveness poll and
`_kickRenderer()`, which does what the dropped `_innerRefresh` would have.
2. Replay clears raced live output. xterm's write() is async-queued while
reset() is synchronous and, per upstream, "does not clear input buffers and
does not reset the parser" — so bytes queued before a reset are parsed after
it and fuse into the snapshot. Verified against the real xterm 6 here:
write('p8'); reset(); write('rmissions') renders "p8rmissions". The main
path was already safe via a queued erase; the needsRefresh and clearTerminal
paths were not. All three now share one queued `\x1bc` (RIS), which unlike
3J/H/2J also resets modes, charsets, scroll regions and SGR state.
3. Output lost on WebSocket reconnect. Input frames carry seq+cid and are
delivered exactly once; output frames carry nothing. ws.onopen re-sends dims
and flushes queued input, and needsRefresh only fires on external-CLI
startup and SSE backpressure drain — never on reconnect. Output produced
while offline was simply absent afterwards. Interim fix: reaching onclose
means the drop was unintentional, so the session is marked and the next open
reconciles from the server buffer. Sequencing output is the follow-up.
4. Terminal captures had no deadline. No AbortController anywhere in the
frontend, including `?full=1`, which the code itself calls "unbounded-ish
work: at the default history limit it can be megabytes". Adds a budget that
scales with full-vs-tail and with captures in flight, degrading to a plain
fetch where AbortController is missing.
Also: the service-worker precache was dead — the build content-hashes assets
but sw.js listed pre-hash names, so 15 of 23 entries 404'd (verified against a
running instance) and cache.add().catch() hid it. Offline still worked via
runtime caching, but CACHE_NAME was a constant so activate's cleanup never
deleted anything and every past release's assets accumulated. Both are now
derived from the build manifest. Crash-trail entries are flattened and capped,
since they are joined with \n into one value and one call site interpolates a
server-controlled WS close reason.
The watchdog reads xterm privates — there is no public API. Every access is
optional-chained so a shape change degrades to a no-op. `_renderService` only
exists after open(), which needs a real DOM, so the gate cannot assert the
field path; test/xterm-private-api.test.ts pins the dependency range instead.
Tests: 23 new (terminal-resilience, sw-precache-manifest, xterm-private-api),
all pure/static so they run in the gate, which excludes the mobile suite. One
static source guard in history-truncation-notice updated for the renamed call;
the behaviour it pins is unchanged.
Not verified: no browser available, so no runtime reproduction of the freeze
and no real-device test of the reconnect path. Both warrant a device pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rounak Datta
co-authored by
Claude Opus 5
Claude Opus 5
parent
9466acfc1a
commit
c0422c4e21
+138
-14
@@ -66,7 +66,18 @@ const _crashDiag = {
|
||||
// concurrent clients (desktop + phone) don't clobber each other.
|
||||
_pageId: Date.now().toString(36) + '-' + Math.random().toString(36).slice(2, 8),
|
||||
log(msg) {
|
||||
const entry = `${new Date().toISOString().slice(11,23)} ${msg}`;
|
||||
// Entries are joined with '\n' into ONE localStorage value and beaconed to
|
||||
// the server, and some call sites interpolate text this client does not
|
||||
// control (a WebSocket close `reason` arrives from the server). A newline
|
||||
// in there forges extra entries in the trail; an unbounded string can fill
|
||||
// the storage quota and silently kill every later breadcrumb. Flatten and
|
||||
// cap. CodemanDiag is loaded before app.js, but guard anyway — a
|
||||
// diagnostic that can throw is worse than no diagnostic.
|
||||
const flat =
|
||||
typeof CodemanDiag !== 'undefined' && CodemanDiag.sanitizeDiagEntry
|
||||
? CodemanDiag.sanitizeDiagEntry(msg)
|
||||
: String(msg == null ? '' : msg).replace(/[\r\n\u2028\u2029]+/g, ' ').slice(0, 300);
|
||||
const entry = `${new Date().toISOString().slice(11,23)} ${flat}`;
|
||||
this._entries.push(entry);
|
||||
if (this._entries.length > this._maxEntries) this._entries.shift();
|
||||
try { localStorage.setItem('codeman-crash-diag', this._entries.join('\n')); } catch {}
|
||||
@@ -682,6 +693,10 @@ class CodemanApp {
|
||||
this._wsReady = false; // True when WS is open and ready for I/O
|
||||
this._wsState = 'disconnected'; // connecting | connected | reconnecting | fallback | disconnected
|
||||
this._wsLastRecvAt = 0; // ms timestamp of the last frame received on the active WS
|
||||
// Session whose socket dropped unintentionally, so output produced during
|
||||
// the outage is missing from its buffer. Output frames carry no sequence
|
||||
// number, so the only recovery is to refetch on the next successful open.
|
||||
this._wsOutputGapSession = null;
|
||||
|
||||
// Terminal write batching with DEC 2026 sync support
|
||||
this.pendingWrites = [];
|
||||
@@ -2555,6 +2570,61 @@ class CodemanApp {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch a terminal capture under a deadline.
|
||||
*
|
||||
* Every terminal fetch used to run with no timeout at all, including
|
||||
* `?full=1`, which _maybeRefetchFullHistory itself calls "unbounded-ish work:
|
||||
* at the default history limit it can be megabytes". On a stalled mobile link
|
||||
* that request hangs on the browser default with no retry, and the load-state
|
||||
* machinery stays armed behind it.
|
||||
*
|
||||
* The budget scales with what is being asked for and with how many captures
|
||||
* are already running (see CodemanFetchDeadline): a full scrollback on a slow
|
||||
* uplink legitimately needs longer than a tail, and eight tabs resuming must
|
||||
* not all expire together because each assumed it had the link to itself.
|
||||
*
|
||||
* An abort surfaces as a rejected fetch, which every caller already handles —
|
||||
* they wrap these in try/catch and log. That is the point: a timeout becomes a
|
||||
* recoverable error instead of an indefinite hang.
|
||||
*
|
||||
* @param {string} url
|
||||
* @param {{full?: boolean}} [opts]
|
||||
* @returns {Promise<Response>}
|
||||
*/
|
||||
async _fetchTerminalCapture(url, opts = {}) {
|
||||
const deadlineMs =
|
||||
typeof CodemanFetchDeadline !== 'undefined'
|
||||
? CodemanFetchDeadline.terminalFetchDeadlineMs({
|
||||
full: !!opts.full,
|
||||
inflight: this._terminalCaptureInflight || 0,
|
||||
})
|
||||
: 45000;
|
||||
// AbortSignal.timeout() is not on every browser Codeman supports, so drive
|
||||
// it from a controller and always clear the timer — an uncancelled one
|
||||
// would abort a LATER request that reused this controller's signal.
|
||||
//
|
||||
// Degrade to a plain fetch where AbortController is missing rather than
|
||||
// throwing: a capture with no deadline is the behaviour every caller had
|
||||
// before this helper existed, while a ReferenceError here would take out
|
||||
// terminal replay entirely. The deadline is a safety net, not a dependency.
|
||||
const canAbort = typeof AbortController === 'function';
|
||||
const controller = canAbort ? new AbortController() : null;
|
||||
const timer = controller ? setTimeout(() => controller.abort(), deadlineMs) : null;
|
||||
this._terminalCaptureInflight = (this._terminalCaptureInflight || 0) + 1;
|
||||
try {
|
||||
return await (controller ? fetch(url, { signal: controller.signal }) : fetch(url));
|
||||
} catch (err) {
|
||||
if (err?.name === 'AbortError') {
|
||||
_crashDiag.log(`TERMINAL FETCH TIMEOUT after ${deadlineMs}ms`);
|
||||
}
|
||||
throw err;
|
||||
} finally {
|
||||
if (timer !== null) clearTimeout(timer);
|
||||
this._terminalCaptureInflight = Math.max(0, (this._terminalCaptureInflight || 1) - 1);
|
||||
}
|
||||
}
|
||||
|
||||
async _onSessionNeedsRefresh(event = {}) {
|
||||
// Server sends this after SSE backpressure clears — terminal data was dropped,
|
||||
// so reload the buffer to recover from any display corruption.
|
||||
@@ -2573,15 +2643,16 @@ class CodemanApp {
|
||||
// TUI modes still recover the whole picture, with the downgrade guard for
|
||||
// repaint-mode panes whose tmux capture can be smaller than xterm's buffer.
|
||||
const useFullHistory = this.sessions.get(sessionId)?.mode !== 'shell';
|
||||
let res = await fetch(
|
||||
let res = await this._fetchTerminalCapture(
|
||||
useFullHistory
|
||||
? `/api/sessions/${sessionId}/terminal?full=1`
|
||||
: `/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`
|
||||
: `/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`,
|
||||
{ full: useFullHistory }
|
||||
);
|
||||
let headersReceivedAt = performance.now();
|
||||
let data = (await res.json())?.data ?? {};
|
||||
if (useFullHistory && data.terminalBuffer && this._replayWouldShrinkBuffer(data.terminalBuffer)) {
|
||||
res = await fetch(`/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`);
|
||||
res = await this._fetchTerminalCapture(`/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`);
|
||||
headersReceivedAt = performance.now();
|
||||
data = (await res.json())?.data ?? {};
|
||||
}
|
||||
@@ -2596,8 +2667,11 @@ class CodemanApp {
|
||||
// meaningless across it — distance from the bottom is what survives.
|
||||
const before = this.terminal.buffer?.active;
|
||||
const linesFromBottom = before ? Math.max(0, (before.baseY || 0) - (before.viewportY || 0)) : 0;
|
||||
this.terminal.clear();
|
||||
this.terminal.reset();
|
||||
// One queued clear, not clear()+reset(): both of those are synchronous
|
||||
// and skip xterm's write queue, so live bytes still parsing would land
|
||||
// after them and fuse into the buffer written below. See
|
||||
// _resetTerminalForReplay.
|
||||
this._resetTerminalForReplay();
|
||||
await this.chunkedTerminalWrite(
|
||||
data.terminalBuffer,
|
||||
TERMINAL_CHUNK_SIZE,
|
||||
@@ -2642,12 +2716,13 @@ class CodemanApp {
|
||||
|
||||
// Fetch buffer, clear terminal, write buffer, resize (no Ctrl+L needed)
|
||||
try {
|
||||
const res = await fetch(`/api/sessions/${data.id}/terminal`);
|
||||
const res = await this._fetchTerminalCapture(`/api/sessions/${data.id}/terminal`);
|
||||
const headersReceivedAt = performance.now();
|
||||
const termData = (await res.json())?.data ?? {};
|
||||
|
||||
this.terminal.clear();
|
||||
this.terminal.reset();
|
||||
// Queued clear — see _resetTerminalForReplay for why clear()+reset()
|
||||
// cannot do this job.
|
||||
this._resetTerminalForReplay();
|
||||
if (termData.terminalBuffer) {
|
||||
// Strip any DEC 2026 markers and write raw content
|
||||
// (markers don't help here - this is a static buffer reload, not live Ink redraws)
|
||||
@@ -2972,6 +3047,18 @@ class CodemanApp {
|
||||
// Flush any durably-queued input over the fresh socket (covers frames a
|
||||
// prior half-open socket silently dropped, and input typed while offline).
|
||||
this._onWsReady(sessionId);
|
||||
// Reconcile the output hole this drop left (see the ws.onclose note).
|
||||
// Only after an unintentional close — a first connect has no gap, and
|
||||
// refetching there would duplicate the buffer selectSession just wrote.
|
||||
if (this._wsOutputGapSession === sessionId) {
|
||||
this._wsOutputGapSession = null;
|
||||
_crashDiag.log(`WS REOPEN: reconciling output gap for ${sessionId}`);
|
||||
// Fire-and-forget: this is recovery, and a failure here must not stop
|
||||
// the socket coming up. _onSessionNeedsRefresh already guards against
|
||||
// running while a buffer load is in flight and against a tab switch
|
||||
// landing this session's history in another session's terminal.
|
||||
void this._onSessionNeedsRefresh({ id: sessionId });
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
@@ -3019,6 +3106,22 @@ class CodemanApp {
|
||||
`WS CLOSE code=${event.code} reason=${event.reason || ''} action=${plan.action} attempts=${this._wsReconnectAttempts || 0}`
|
||||
);
|
||||
|
||||
// Output frames carry no sequence number, so a socket that dropped left a
|
||||
// hole in the terminal with nothing to replay it: ws.onopen re-sends dims
|
||||
// and flushes queued INPUT, and `needsRefresh` only fires on external-CLI
|
||||
// startup and on SSE backpressure drain — never here. Whatever the PTY
|
||||
// produced while the link was down is simply absent from the buffer.
|
||||
//
|
||||
// Reaching onclose at all means the drop was NOT intentional
|
||||
// (_disconnectWs nulls this handler first), so mark the gap and let the
|
||||
// next successful open reconcile from the server's buffer.
|
||||
//
|
||||
// Scoped to the session that actually lost bytes: a user who switches
|
||||
// sessions during an outage gets a clean intentional disconnect for the
|
||||
// new one, and its freshly-loaded buffer must not be refetched because a
|
||||
// DIFFERENT session's socket dropped.
|
||||
this._wsOutputGapSession = sessionId;
|
||||
|
||||
const stillActive = this.activeSessionId === sessionId;
|
||||
if (plan.action === 'give-up') {
|
||||
this._wsState = stillActive ? 'fallback' : 'disconnected';
|
||||
@@ -5866,9 +5969,29 @@ class CodemanApp {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear the terminal for a replay, IN STREAM.
|
||||
*
|
||||
* xterm's `write()` is asynchronously queued (the WriteBuffer parses in ~12ms
|
||||
* slices) while `Terminal.reset()` is synchronous and, by upstream's own
|
||||
* documentation, "does not clear input buffers and does not reset the parser,
|
||||
* thus the terminal will continue to apply pending input data". So bytes
|
||||
* queued just before a `reset()` are parsed AFTER it and fuse into whatever
|
||||
* snapshot is written next — measured upstream as `p8rmissions` rendered
|
||||
* where `bypass permissions` belonged.
|
||||
*
|
||||
* A queued clear cannot race that way: it lands after the leftovers and
|
||||
* before the snapshot, whatever the queue held. This function used to follow
|
||||
* the sync `reset()` with a queued `\x1b[3J\x1b[H\x1b[2J`, which already got
|
||||
* that right for CONTENT. RIS (`\x1bc`) additionally resets modes, charsets,
|
||||
* scroll regions and SGR state, so leftover bytes cannot park the terminal in
|
||||
* alt-screen or an odd scroll region and survive the clear.
|
||||
*
|
||||
* Callers may write the replacement content in as many chunks as they like —
|
||||
* ordering within the queue is what matters, not writing it all at once.
|
||||
*/
|
||||
_resetTerminalForReplay() {
|
||||
this.terminal.reset();
|
||||
this.terminal.write('\x1b[3J\x1b[H\x1b[2J');
|
||||
this.terminal.write('\x1bc');
|
||||
}
|
||||
|
||||
_recordTerminalLoadTiming(timing) {
|
||||
@@ -5932,7 +6055,7 @@ class CodemanApp {
|
||||
this._fullHistoryRepullInFlight = true;
|
||||
try {
|
||||
const requestStartedAt = performance.now();
|
||||
const res = await fetch(`/api/sessions/${sessionId}/terminal?full=1`);
|
||||
const res = await this._fetchTerminalCapture(`/api/sessions/${sessionId}/terminal?full=1`, { full: true });
|
||||
const headersReceivedAt = performance.now();
|
||||
const payload = (await res.json())?.data ?? {};
|
||||
const bodyParsedAt = performance.now();
|
||||
@@ -6445,10 +6568,11 @@ class CodemanApp {
|
||||
const useFullHistory = session?.mode !== 'shell' && !this._fullHistoryLoaded.has(sessionId);
|
||||
if (useFullHistory) this._fullHistoryLoaded.add(sessionId);
|
||||
const fetchStartedAt = performance.now();
|
||||
const res = await fetch(
|
||||
const res = await this._fetchTerminalCapture(
|
||||
useFullHistory
|
||||
? `/api/sessions/${sessionId}/terminal?full=1`
|
||||
: `/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`
|
||||
: `/api/sessions/${sessionId}/terminal?tail=${TERMINAL_TAIL_SIZE}`,
|
||||
{ full: useFullHistory }
|
||||
);
|
||||
const headersReceivedAt = performance.now();
|
||||
if (this._isStaleSelect(selectGen)) {
|
||||
|
||||
Reference in New Issue
Block a user