mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-09-30 12:39:42 +02:00
Merge branch 'fix/sse-stale-watchdog'
Heal a stalled SSE stream: the server's :keepalive comment becomes a named sse:heartbeat event (comments are invisible to EventSource by spec), and the client gains a staleness watchdog that forces a reconnect after three missed beats. Also applies a confirmed rename locally instead of waiting on SSE. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -292,6 +292,8 @@ Frontend JS modules have `@fileoverview` with `@dependency`/`@loadorder` tags. L
|
||||
|
||||
**Connection-loss UI** (`computeConnectionLossUi()` in constants.js, writer `_updateConnectionLossUi()` in app.js): the service worker serves the cached app shell, so an unreachable server (phone off the tailnet, VPN down, server stopped) used to render a normal-looking empty dashboard whose only tell was the 8px header dot, which reads as "no sessions", not "no connection". Two surfaces now: a full-screen **overlay** while no server state has loaded this page load (nothing behind it is worth preserving), and a non-blocking **banner** once it has (the terminal scrollback stays readable). ⚠️ A **2.5s grace** is load-bearing: a COM deploy restarts the server and SSE is back in ~200ms, and a banner on every deploy trains the user to ignore it. `navigator.onLine === false` skips the grace, since that is never a blip. Retry re-arms SSE **and** the terminal WS (`planWsReconnect` can 'give-up', and the SSE backoff caps at 30s).
|
||||
|
||||
**SSE staleness watchdog** (`computeSseStale()` in constants.js, `_checkSseStale()` + a 5s interval in app.js): an `EventSource` that stops delivering does not always error, so `onerror` never fires, the header dot stays green, and every SSE-driven surface (tab status dots, sessions created on another device, renames) freezes until the user reloads. ⚠️ The 15s server keepalive was an SSE **comment** (`:keepalive`), and comments are **invisible to `EventSource` by spec**, so there was nothing a client could observe: it is now the named `sse:heartbeat` event (`cleanupDeadClients()`, sse-stream-manager.ts), which is exactly why the frame had to change type. ⚠️ Staleness is judged **only while the status is `connected`** and the device is online; that guard is the loop breaker, since a forced `connectSSE()` leaves `connected` immediately and cannot re-fire while a reconnect is in flight. ⚠️ The liveness stamp is applied inside `addListener` itself, so every registered handler (the `_SSE_HANDLER_MAP` wrappers AND the directly-registered ones) feeds it from one place; the heartbeat's own listener is a no-op that exists **only** to be registered, since `EventSource` drops named events nobody listens for. ⚠️ The watchdog interval is cleared at the top of `connectSSE()` and nowhere else (its only teardown path); clearing it elsewhere stacks intervals. Recovery needs no new sync path: the reconnect re-runs `handleInit` → `_resetAllAppState()`. The forced reconnect logs one diagnostic line, because a middlebox that strips heartbeats presents as "silently reconnects every 45s".
|
||||
|
||||
**Z-index layers**: subagent windows (1000), plan agents (1100), mobile/tablet fixed header (1200, `mobile.css`), modals on ≤768px (1300 — must beat the fixed header or the modal close button is buried), log viewers (2000), connection-loss overlay (2500, above the fixed header and modals), image popups (3000), local echo overlay (7).
|
||||
|
||||
**Respawn presets**: `solo-work` (3s/60min), `subagent-workflow` (45s/240min), `team-lead` (90s/480min), `ralph-todo` (8s/480min), `overnight-autonomous` (10s/480min).
|
||||
@@ -320,7 +322,7 @@ Frontend JS modules have `@fileoverview` with `@dependency`/`@loadorder` tags. L
|
||||
|
||||
### SSE Event Registry
|
||||
|
||||
154 event constants in `src/web/sse-events.ts` (backend) and `SSE_EVENTS` in `constants.js` (frontend). **Both must be kept in sync**, and `test/sse-registry-parity.test.ts` is the guard that pins it (currently exactly in sync, 154 = 154, no drift either direction). The backend file's `@fileoverview` carries the per-category breakdown.
|
||||
155 event constants in `src/web/sse-events.ts` (backend) and `SSE_EVENTS` in `constants.js` (frontend). **Both must be kept in sync**, and `test/sse-registry-parity.test.ts` is the guard that pins it (currently exactly in sync, 155 = 155, no drift either direction). The backend file's `@fileoverview` carries the per-category breakdown.
|
||||
|
||||
### API Routes
|
||||
|
||||
|
||||
@@ -534,6 +534,29 @@ the stable contract — event names are not renamed without a major bump. An
|
||||
optional `?sessions=<id,...>` filter suppresses only the high-volume terminal
|
||||
stream; lifecycle/metadata events are delivered to all clients regardless.
|
||||
|
||||
### `sse:heartbeat` (liveness)
|
||||
|
||||
Every 15s the server writes a `sse:heartbeat` frame to every connected client:
|
||||
|
||||
```
|
||||
event: sse:heartbeat
|
||||
data: {"t":1755100000000}
|
||||
```
|
||||
|
||||
`t` is the server's epoch-ms timestamp at write time. The frame carries no
|
||||
application state and can be ignored for correctness. It exists so a client can
|
||||
tell a live stream from a dead one: an `EventSource` whose connection has been
|
||||
idle-closed by a proxy (or that resumed from sleep on a stale socket) keeps
|
||||
delivering nothing without ever firing `onerror`. Clients that care should treat
|
||||
silence longer than about three intervals as a dead stream and reconnect, which
|
||||
is what the bundled frontend does.
|
||||
|
||||
This replaced a `:keepalive` SSE **comment**, which served the same
|
||||
proxy-flushing purpose but is invisible to `EventSource` by spec and so could
|
||||
never be observed by a client. Consumers written against the old behavior are
|
||||
unaffected: `EventSource` dispatches only events that have a registered
|
||||
listener, so an unknown event name is dropped.
|
||||
|
||||
## Consuming from JavaScript
|
||||
|
||||
The bundled frontend reads responses through `_apiJson()`
|
||||
|
||||
+86
-3
@@ -684,6 +684,17 @@ class CodemanApp {
|
||||
this.maxReconnectAttempts = 10;
|
||||
this.isOnline = navigator.onLine;
|
||||
|
||||
// SSE staleness watchdog. An EventSource that stops delivering does not
|
||||
// always error (a proxy that idle-closed it, a resumed laptop), so
|
||||
// `onerror` never fires and every SSE-driven surface freezes silently.
|
||||
// The server heartbeats every 15s; going quiet for three of them means the
|
||||
// stream is a zombie and has to be rebuilt. The decision is pure
|
||||
// (computeSseStale in constants.js); these are its inputs. The threshold
|
||||
// is an instance field so a browser test can shrink it.
|
||||
this._sseLastMessageAt = 0;
|
||||
this._sseStaleTimeoutMs = window.CodemanSseStale?.TIMEOUT_MS ?? 45000;
|
||||
this._sseStaleWatchdog = null;
|
||||
|
||||
// Connection-loss UI (banner + full-screen overlay). The decision itself is
|
||||
// pure and lives in constants.js (computeConnectionLossUi); these are just
|
||||
// its inputs. `_connDownSince` is the timestamp the transport LEFT the
|
||||
@@ -719,6 +730,11 @@ class CodemanApp {
|
||||
window.addEventListener('pagehide', () => this._persistReliableNow());
|
||||
document.addEventListener('visibilitychange', () => {
|
||||
if (document.visibilityState === 'hidden') this._persistReliableNow();
|
||||
// A background tab's timers are throttled, so the 5s watchdog may not
|
||||
// have run for minutes, and a wake/unlock is exactly when a stream
|
||||
// comes back zombie. Checking here is what makes recovery feel instant
|
||||
// instead of up to a full timeout late.
|
||||
else this._checkSseStale();
|
||||
});
|
||||
|
||||
// Local echo overlay — DOM overlay positioned at the visible ❯ prompt
|
||||
@@ -1421,6 +1437,14 @@ class CodemanApp {
|
||||
// Clear any pending reconnect timeout to prevent duplicate connections
|
||||
this._clearTimer('sseReconnectTimeout');
|
||||
|
||||
// Same discipline for the staleness watchdog: connectSSE() runs on every
|
||||
// reconnect and is the only teardown path this page-lifetime interval has,
|
||||
// so clearing it anywhere else (or not at all) stacks intervals.
|
||||
if (this._sseStaleWatchdog) {
|
||||
clearInterval(this._sseStaleWatchdog);
|
||||
this._sseStaleWatchdog = null;
|
||||
}
|
||||
|
||||
// Clean up existing SSE listeners before creating new connection (prevents listener accumulation)
|
||||
if (this._sseListenerCleanup) {
|
||||
this._sseListenerCleanup();
|
||||
@@ -1448,11 +1472,20 @@ class CodemanApp {
|
||||
if (this.activeSessionId) _sseParams.set('sessions', this.activeSessionId);
|
||||
this.eventSource = new EventSource(`/api/events?${_sseParams.toString()}`);
|
||||
|
||||
// Store all event listeners for cleanup on reconnect
|
||||
// Store all event listeners for cleanup on reconnect.
|
||||
//
|
||||
// Every handler is wrapped so ANY frame that arrives stamps the liveness
|
||||
// clock the staleness watchdog reads. Doing it here (rather than at the
|
||||
// three separate registration sites below) is what keeps a future
|
||||
// addListener() call from silently opting out of it.
|
||||
const listeners = [];
|
||||
const addListener = (event, handler) => {
|
||||
this.eventSource.addEventListener(event, handler);
|
||||
listeners.push({ event, handler });
|
||||
const stamped = (e) => {
|
||||
this._sseLastMessageAt = Date.now();
|
||||
handler(e);
|
||||
};
|
||||
this.eventSource.addEventListener(event, stamped);
|
||||
listeners.push({ event, handler: stamped });
|
||||
};
|
||||
|
||||
// Create cleanup function to remove all listeners
|
||||
@@ -1467,6 +1500,10 @@ class CodemanApp {
|
||||
|
||||
this.eventSource.onopen = () => {
|
||||
this.reconnectAttempts = 0;
|
||||
// Start the liveness clock here, not at the first frame: the watchdog
|
||||
// only ever fires while the status is 'connected', and this is the
|
||||
// moment that becomes true.
|
||||
this._sseLastMessageAt = Date.now();
|
||||
this.setConnectionStatus('connected');
|
||||
};
|
||||
this.eventSource.onerror = () => {
|
||||
@@ -1614,6 +1651,52 @@ class CodemanApp {
|
||||
}
|
||||
this._onSessionListMaybeChanged();
|
||||
});
|
||||
|
||||
// Liveness heartbeat. The handler is deliberately empty: the whole point
|
||||
// is the stamp inherited from addListener's wrapper. It still has to be
|
||||
// REGISTERED: EventSource only dispatches named events that have a
|
||||
// listener, so without this the frame arrives on the wire and is dropped
|
||||
// before it can prove the stream is alive.
|
||||
addListener(SSE_EVENTS.HEARTBEAT, () => {});
|
||||
|
||||
// Watchdog: a stream that goes quiet without erroring is invisible to
|
||||
// onerror, so poll the pure staleness policy and rebuild the connection
|
||||
// ourselves. 5s granularity against a 45s threshold: cheap, and it keeps
|
||||
// the worst-case detection lag well under a heartbeat interval.
|
||||
this._sseStaleWatchdog = setInterval(() => this._checkSseStale(), 5000);
|
||||
}
|
||||
|
||||
/**
|
||||
* Force a reconnect if the SSE stream has gone quiet while still claiming to
|
||||
* be connected. Called by the 5s watchdog and on tab-visible.
|
||||
*
|
||||
* Recovery needs no new sync path: the reconnect re-runs `handleInit`, which
|
||||
* already calls `_resetAllAppState()` and rebuilds everything from the
|
||||
* server. The connection-loss UI needs nothing either: `connectSSE()` sets
|
||||
* status 'connecting' (reconnectAttempts was zeroed by onopen), and the 2.5s
|
||||
* grace in computeConnectionLossUi means a stream that heals in 200ms shows
|
||||
* nothing at all.
|
||||
*/
|
||||
_checkSseStale() {
|
||||
const policy = window.CodemanSseStale;
|
||||
if (!policy) return;
|
||||
const now = Date.now();
|
||||
const stale = policy.compute({
|
||||
lastMessageAt: this._sseLastMessageAt,
|
||||
now,
|
||||
status: this._connectionStatus,
|
||||
isOnline: this.isOnline,
|
||||
timeoutMs: this._sseStaleTimeoutMs,
|
||||
});
|
||||
if (!stale) return;
|
||||
// If a middlebox ever strips or delays heartbeats, the failure mode is
|
||||
// "silently reconnects every 45s", and a field report of that would be
|
||||
// undebuggable without this line.
|
||||
console.log(
|
||||
`[SSE] stream stale: no frame for ${now - this._sseLastMessageAt}ms ` +
|
||||
`(threshold ${this._sseStaleTimeoutMs}ms), forcing reconnect`
|
||||
);
|
||||
this.connectSSE();
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════
|
||||
|
||||
@@ -379,6 +379,41 @@ function computeConnectionLossUi(input) {
|
||||
};
|
||||
}
|
||||
|
||||
// SSE staleness policy: is this stream a zombie?
|
||||
//
|
||||
// An EventSource that stops delivering does not always error. A proxy that
|
||||
// idle-closed the connection, a laptop resumed from sleep, a tailnet
|
||||
// reconnect: `onerror` never fires, the header dot stays green, and every
|
||||
// SSE-driven surface (tab status dots, sessions created on another device,
|
||||
// renames) freezes until the user reloads. The server writes a
|
||||
// `sse:heartbeat` frame every 15s, so silence longer than three of them means
|
||||
// the stream is dead even though the transport still claims otherwise.
|
||||
//
|
||||
// Stale ONLY when the transport believes it is 'connected': the other states
|
||||
// already have the reconnect/backoff machinery running, and re-firing on top
|
||||
// of them would stack reconnects. That guard is also the loop breaker: a
|
||||
// forced reconnect leaves 'connected' immediately, so the watchdog cannot
|
||||
// fire again while one is in flight. `navigator.onLine === false` is not
|
||||
// staleness either; there is nothing to reconnect to yet.
|
||||
//
|
||||
// Pure: no DOM, no timers, no side effects. `now` is passed in.
|
||||
const SSE_STALE_TIMEOUT_MS = 45000; // three missed 15s heartbeats
|
||||
|
||||
function computeSseStale(input) {
|
||||
const {
|
||||
lastMessageAt = null,
|
||||
now = 0,
|
||||
status = 'connected',
|
||||
isOnline = true,
|
||||
timeoutMs = SSE_STALE_TIMEOUT_MS,
|
||||
} = input || {};
|
||||
if (!isOnline || status !== 'connected') return false;
|
||||
// No frame has ever arrived: `init` lands on connect, so this is a stream
|
||||
// that has not opened yet rather than one that went quiet.
|
||||
if (typeof lastMessageAt !== 'number' || !(lastMessageAt > 0)) return false;
|
||||
return now - lastMessageAt >= timeoutMs;
|
||||
}
|
||||
|
||||
if (typeof window !== 'undefined') {
|
||||
window.WEBGL_FALLBACK = WEBGL_FALLBACK;
|
||||
window.evaluateWebGLLongTaskTrip = evaluateWebGLLongTaskTrip;
|
||||
@@ -401,6 +436,10 @@ if (typeof window !== 'undefined') {
|
||||
compute: computeConnectionLossUi,
|
||||
GRACE_MS: CONNECTION_LOSS_GRACE_MS,
|
||||
};
|
||||
window.CodemanSseStale = {
|
||||
compute: computeSseStale,
|
||||
TIMEOUT_MS: SSE_STALE_TIMEOUT_MS,
|
||||
};
|
||||
}
|
||||
|
||||
// Scheduler API — prioritize terminal writes over background UI updates.
|
||||
@@ -514,6 +553,9 @@ const SSE_EVENTS = {
|
||||
// Core
|
||||
INIT: 'init',
|
||||
|
||||
// Transport
|
||||
HEARTBEAT: 'sse:heartbeat',
|
||||
|
||||
// Session lifecycle
|
||||
SESSION_CREATED: 'session:created',
|
||||
SESSION_UPDATED: 'session:updated',
|
||||
|
||||
@@ -1415,9 +1415,54 @@ Object.assign(CodemanApp.prototype, {
|
||||
this.activeFocusTrap.activate();
|
||||
},
|
||||
|
||||
/**
|
||||
* Write a name the server has just confirmed into the local session map.
|
||||
*
|
||||
* Both rename surfaces re-render the tab strip from `this.sessions` right
|
||||
* after their PUT, so without this they depended on the `session:updated` SSE
|
||||
* frame to carry their own write back. On a page whose SSE stream has gone
|
||||
* quiet without erroring (a proxy that idle-closed it, a laptop resumed from
|
||||
* sleep) that frame never lands: the PUT stores the new name, the re-render
|
||||
* repaints the stale one, and the rename looks like it did nothing until a
|
||||
* full page reload. The response body is authoritative, so apply it directly.
|
||||
* The SSE frame, when it does arrive, replaces the object with the same name.
|
||||
*/
|
||||
_applyLocalSessionName(sessionId, name) {
|
||||
if (typeof name !== 'string') return;
|
||||
const session = this.sessions.get(sessionId);
|
||||
if (!session) return;
|
||||
session.name = name;
|
||||
this.sessions.set(sessionId, session);
|
||||
// Mirrors _onSessionUpdated: subagent windows cache their parent's name.
|
||||
this.updateSubagentParentNames?.(sessionId);
|
||||
},
|
||||
|
||||
/**
|
||||
* PUT a session name and return the name the server stored, or null if the
|
||||
* request failed. `_apiPut` swallows network errors into a null Response and
|
||||
* an API-level failure arrives as a non-ok status or `{success:false}`, so a
|
||||
* rename that silently did nothing has to be detected here, not thrown.
|
||||
*/
|
||||
async _putSessionName(sessionId, name) {
|
||||
const res = await this._apiPut(`/api/sessions/${sessionId}/name`, { name });
|
||||
if (!res || !res.ok) return null;
|
||||
let payload = null;
|
||||
try {
|
||||
payload = await res.json();
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (payload && payload.success === false) return null;
|
||||
const confirmed = payload?.data?.name;
|
||||
return typeof confirmed === 'string' ? confirmed : name;
|
||||
},
|
||||
|
||||
async saveSessionName() {
|
||||
if (!this.editingSessionId) return;
|
||||
const session = this.sessions.get(this.editingSessionId);
|
||||
// Captured: the modal can be closed (or switched to another session) while
|
||||
// the PUT is in flight, and the name belongs to the session that was open.
|
||||
const sessionId = this.editingSessionId;
|
||||
const session = this.sessions.get(sessionId);
|
||||
const parsed = session ? parseSessionPrefix(session.name) : null;
|
||||
const inputVal = document.getElementById('modalSessionName').value.trim();
|
||||
let name;
|
||||
@@ -1426,11 +1471,13 @@ Object.assign(CodemanApp.prototype, {
|
||||
} else {
|
||||
name = inputVal;
|
||||
}
|
||||
try {
|
||||
await this._apiPut(`/api/sessions/${this.editingSessionId}/name`, { name });
|
||||
} catch (err) {
|
||||
this.showToast('Failed to save session name: ' + err.message, 'error');
|
||||
const confirmed = await this._putSessionName(sessionId, name);
|
||||
if (confirmed === null) {
|
||||
this.showToast('Failed to save session name', 'error');
|
||||
return;
|
||||
}
|
||||
this._applyLocalSessionName(sessionId, confirmed);
|
||||
this.renderSessionTabs();
|
||||
},
|
||||
|
||||
async autoSaveAutoCompact() {
|
||||
@@ -1740,15 +1787,14 @@ Object.assign(CodemanApp.prototype, {
|
||||
// Skip the API call if the session vanished between focus and blur.
|
||||
const stillExists = this.sessions.has(sessionId);
|
||||
if (stillExists && fullName !== session.name) {
|
||||
try {
|
||||
await fetch(`/api/sessions/${sessionId}/name`, {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ name: fullName })
|
||||
});
|
||||
} catch (err) {
|
||||
const confirmed = await this._putSessionName(sessionId, fullName);
|
||||
if (confirmed === null) {
|
||||
tabName.textContent = originalContent;
|
||||
this.showToast('Failed to rename', 'error');
|
||||
} else {
|
||||
// The re-render below repaints from this.sessions, so the new name has
|
||||
// to be in the map before it runs (see _applyLocalSessionName()).
|
||||
this._applyLocalSessionName(sessionId, confirmed);
|
||||
}
|
||||
}
|
||||
// Re-render tabs to restore full tab structure
|
||||
|
||||
+21
-1
@@ -5,8 +5,9 @@
|
||||
* and referenced by the frontend (`SSE_EVENTS` in `constants.js`).
|
||||
* Both files MUST be kept in sync.
|
||||
*
|
||||
* 154 event constants organized by category:
|
||||
* 155 event constants organized by category:
|
||||
* - **Core** (1): init
|
||||
* - **Transport** (1): sse:heartbeat
|
||||
* - **Session lifecycle** (23): created, updated, deleted, terminal, idle, working, ...
|
||||
* - **Session: Ralph** (6): ralphLoopUpdate, todoUpdate, completionDetected, ...
|
||||
* - **Session: Bash tools** (3): bashToolStart, bashToolEnd, bashToolsUpdate
|
||||
@@ -52,6 +53,22 @@
|
||||
/** Sent to each SSE client on initial connection with full app state. */
|
||||
export const Init = 'init' as const;
|
||||
|
||||
// ─── Transport ───────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Liveness frame written to every SSE client every `SSE_HEARTBEAT_INTERVAL`.
|
||||
* Payload: `{ t: <epoch ms> }`.
|
||||
*
|
||||
* Carries no application data; its only job is to be *observable*. This was a
|
||||
* `:keepalive` SSE **comment**, and comments are invisible to `EventSource` by
|
||||
* spec, so a stream that stopped delivering without erroring (a proxy that
|
||||
* idle-closed it, a laptop resumed from sleep, a tailnet reconnect) was
|
||||
* undetectable to the client: `onerror` never fires and the UI freezes until a
|
||||
* reload. A named event reaches a listener, which is what lets the client's
|
||||
* staleness watchdog notice the silence and force a reconnect.
|
||||
*/
|
||||
export const Heartbeat = 'sse:heartbeat' as const;
|
||||
|
||||
// ─── Session Lifecycle ───────────────────────────────────────────────────────
|
||||
|
||||
/** New session spawned. */
|
||||
@@ -443,6 +460,9 @@ export const SseEvent = {
|
||||
// Core
|
||||
Init,
|
||||
|
||||
// Transport
|
||||
Heartbeat,
|
||||
|
||||
// Session lifecycle
|
||||
SessionCreated,
|
||||
SessionUpdated,
|
||||
|
||||
@@ -470,12 +470,20 @@ export class SseStreamManager {
|
||||
// ========== Client Health ==========
|
||||
|
||||
/**
|
||||
* Clean up dead SSE clients and send keep-alive comments.
|
||||
* Clean up dead SSE clients and send the liveness heartbeat.
|
||||
* Keep-alive prevents proxy/load-balancer timeouts on idle connections.
|
||||
* Dead client cleanup prevents memory leaks from abruptly terminated connections.
|
||||
*
|
||||
* The heartbeat is a NAMED event, not the `:keepalive` comment it used to be:
|
||||
* comments are invisible to `EventSource` by spec, so a stream that stopped
|
||||
* delivering without erroring was undetectable to the client (see
|
||||
* `SseEvent.Heartbeat`). Written per-client rather than through `broadcast()`
|
||||
* deliberately: the frame carries no session data, so it needs no owner
|
||||
* routing, and this loop is already walking every client to check its socket.
|
||||
*/
|
||||
cleanupDeadClients(): void {
|
||||
const deadClients: FastifyReply[] = [];
|
||||
const heartbeat = `event: ${SseEvent.Heartbeat}\ndata: ${JSON.stringify({ t: Date.now() })}\n\n`;
|
||||
|
||||
for (const [client] of this.sseClients) {
|
||||
try {
|
||||
@@ -484,11 +492,9 @@ export class SseStreamManager {
|
||||
if (!socket || socket.destroyed || !socket.writable) {
|
||||
deadClients.push(client);
|
||||
} else {
|
||||
// Send SSE comment as keep-alive. Only add padding when tunnel is
|
||||
// active — it flushes Cloudflare proxy buffers but wastes bandwidth
|
||||
// for direct/Tailscale connections.
|
||||
const ka = this._isTunnelActive ? ':keepalive\n' + SSE_PADDING : ':keepalive\n\n';
|
||||
client.raw.write(ka);
|
||||
// Only add padding when tunnel is active: it flushes Cloudflare
|
||||
// proxy buffers but wastes bandwidth for direct/Tailscale connections.
|
||||
client.raw.write(this._isTunnelActive ? heartbeat + SSE_PADDING : heartbeat);
|
||||
}
|
||||
} catch {
|
||||
// Error accessing socket means client is dead
|
||||
|
||||
@@ -358,6 +358,83 @@ describe('Inline rename input', () => {
|
||||
expect(result.editingAfter).toBe(null);
|
||||
});
|
||||
|
||||
it('Commit writes the confirmed name into app.sessions WITHOUT any session:updated frame', async () => {
|
||||
await resetState();
|
||||
expect(await startRename('no-sse', 'w9-case')).toBe(true);
|
||||
|
||||
// finishRename() re-renders the tab strip from app.sessions, so the rename
|
||||
// used to depend on the session:updated SSE frame to carry its own write
|
||||
// back. On a page whose stream has gone quiet without erroring, the PUT
|
||||
// stored the new name, the re-render repainted the stale one, and the tab
|
||||
// only showed it after a full reload. No SSE is dispatched here at all.
|
||||
const result = await page.evaluate(async () => {
|
||||
const app = (
|
||||
window as unknown as {
|
||||
app: { sessions: Map<string, { id: string; name: string }> };
|
||||
}
|
||||
).app;
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = (async () =>
|
||||
new Response('{"success":true,"data":{"name":"w9-case: fresh"}}', {
|
||||
status: 200,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
})) as typeof window.fetch;
|
||||
|
||||
const inputEl = document.querySelector('input.tab-rename-input') as HTMLInputElement;
|
||||
inputEl.value = 'fresh';
|
||||
inputEl.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true }));
|
||||
await new Promise((r) => setTimeout(r, 60));
|
||||
|
||||
window.fetch = origFetch;
|
||||
return { mapName: app.sessions.get('no-sse')?.name ?? null };
|
||||
});
|
||||
|
||||
expect(result.mapName).toBe('w9-case: fresh');
|
||||
});
|
||||
|
||||
it('A rejected rename restores the old label and leaves app.sessions untouched', async () => {
|
||||
await resetState();
|
||||
expect(await startRename('rename-500', 'w9-case')).toBe(true);
|
||||
|
||||
// _apiPut turns a network error into a null Response and an API-level
|
||||
// failure arrives as a non-ok status, neither of which throws, so a
|
||||
// rejected rename has to be detected from the response, or it reports
|
||||
// success and silently discards the user's edit.
|
||||
const result = await page.evaluate(async () => {
|
||||
const app = (
|
||||
window as unknown as {
|
||||
app: { sessions: Map<string, { id: string; name: string }>; showToast: (m: string, k: string) => void };
|
||||
}
|
||||
).app;
|
||||
const toasts: string[] = [];
|
||||
const origToast = app.showToast;
|
||||
app.showToast = (msg: string) => void toasts.push(msg);
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = (async () =>
|
||||
new Response('{"success":false,"error":"boom","errorCode":"INTERNAL"}', {
|
||||
status: 500,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
})) as typeof window.fetch;
|
||||
|
||||
const inputEl = document.querySelector('input.tab-rename-input') as HTMLInputElement;
|
||||
inputEl.value = 'never-stored';
|
||||
inputEl.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true }));
|
||||
await new Promise((r) => setTimeout(r, 60));
|
||||
|
||||
window.fetch = origFetch;
|
||||
app.showToast = origToast;
|
||||
return {
|
||||
mapName: app.sessions.get('rename-500')?.name ?? null,
|
||||
label: document.querySelector('.tab-name[data-session-id="rename-500"]')?.textContent ?? null,
|
||||
toasts,
|
||||
};
|
||||
});
|
||||
|
||||
expect(result.mapName).toBe('w9-case');
|
||||
expect(result.label).toBe('w9-case');
|
||||
expect(result.toasts).toContain('Failed to rename');
|
||||
});
|
||||
|
||||
it('Re-entry: starting rename while one is active aborts the previous one', async () => {
|
||||
await resetState();
|
||||
expect(await startRename('first-id', 'First')).toBe(true);
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
/**
|
||||
* SSE liveness heartbeat.
|
||||
*
|
||||
* `cleanupDeadClients()` runs every SSE_HEARTBEAT_INTERVAL (15s) and does two
|
||||
* jobs: evict clients whose socket died, and write a liveness frame to the
|
||||
* ones that are still up.
|
||||
*
|
||||
* The regression these guard: that frame used to be an SSE `:keepalive`
|
||||
* COMMENT, and comments are invisible to `EventSource` by spec. A stream that
|
||||
* stopped delivering without erroring was therefore undetectable to the
|
||||
* client: `onerror` never fired, the header dot stayed green, and every
|
||||
* SSE-driven surface froze until the user reloaded. A named `sse:heartbeat`
|
||||
* event reaches a listener, which is what lets the client's staleness
|
||||
* watchdog notice the silence (see test/sse-staleness.test.ts).
|
||||
*
|
||||
* No port needed (the manager is driven directly with fake replies).
|
||||
*/
|
||||
import type { FastifyReply } from 'fastify';
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { SSE_PADDING_SIZE } from '../src/config/server-timing.js';
|
||||
import { SseEvent } from '../src/web/sse-events.js';
|
||||
import { SseStreamManager } from '../src/web/sse-stream-manager.js';
|
||||
import { CleanupManager } from '../src/utils/index.js';
|
||||
|
||||
/** A FastifyReply stand-in that records every raw write. */
|
||||
function fakeClient(opts: { destroyed?: boolean; writable?: boolean; throwOnAccess?: boolean } = {}) {
|
||||
const writes: string[] = [];
|
||||
const socket = { destroyed: opts.destroyed ?? false, writable: opts.writable ?? true };
|
||||
const raw = {
|
||||
get socket() {
|
||||
if (opts.throwOnAccess) throw new Error('socket gone');
|
||||
return socket;
|
||||
},
|
||||
write(chunk: string) {
|
||||
writes.push(chunk);
|
||||
return true;
|
||||
},
|
||||
};
|
||||
return { reply: { raw } as unknown as FastifyReply, writes };
|
||||
}
|
||||
|
||||
function makeManager() {
|
||||
const cleanup = new CleanupManager();
|
||||
const manager = new SseStreamManager({ getSessionStateWithRespawn: () => null }, cleanup);
|
||||
return { manager, cleanup };
|
||||
}
|
||||
|
||||
describe('SSE liveness heartbeat', () => {
|
||||
it('writes a NAMED sse:heartbeat event, not an invisible comment', () => {
|
||||
const { manager, cleanup } = makeManager();
|
||||
const client = fakeClient();
|
||||
manager.addClient(client.reply, null, false);
|
||||
|
||||
manager.cleanupDeadClients();
|
||||
|
||||
expect(client.writes).toHaveLength(1);
|
||||
const frame = client.writes[0];
|
||||
// A comment (`:keepalive`) never reaches an EventSource listener, and that is
|
||||
// the entire bug. The frame must be a dispatchable named event.
|
||||
expect(frame.startsWith(':')).toBe(false);
|
||||
expect(frame).toMatch(/^event: sse:heartbeat\n/);
|
||||
expect(frame.endsWith('\n\n')).toBe(true);
|
||||
expect(SseEvent.Heartbeat).toBe('sse:heartbeat');
|
||||
cleanup.dispose();
|
||||
});
|
||||
|
||||
it('carries a parseable epoch-ms payload', () => {
|
||||
const { manager, cleanup } = makeManager();
|
||||
const client = fakeClient();
|
||||
manager.addClient(client.reply, null, false);
|
||||
const before = Date.now();
|
||||
|
||||
manager.cleanupDeadClients();
|
||||
|
||||
const dataLine = client.writes[0].split('\n').find((l) => l.startsWith('data: '));
|
||||
expect(dataLine).toBeDefined();
|
||||
const payload = JSON.parse(dataLine!.slice('data: '.length)) as { t: number };
|
||||
expect(payload.t).toBeGreaterThanOrEqual(before);
|
||||
expect(payload.t).toBeLessThanOrEqual(Date.now());
|
||||
cleanup.dispose();
|
||||
});
|
||||
|
||||
it('still appends Cloudflare tunnel padding when a tunnel is active', () => {
|
||||
const { manager, cleanup } = makeManager();
|
||||
const client = fakeClient();
|
||||
manager.addClient(client.reply, null, false);
|
||||
manager.setTunnelActive(true);
|
||||
|
||||
manager.cleanupDeadClients();
|
||||
|
||||
const frame = client.writes[0];
|
||||
expect(frame).toMatch(/^event: sse:heartbeat\n/);
|
||||
// Padding rides AFTER the terminating blank line, so the event still parses.
|
||||
const [event, padding] = frame.split('\n\n');
|
||||
expect(event).toMatch(/^event: sse:heartbeat\ndata: \{/);
|
||||
expect(padding.startsWith(':')).toBe(true);
|
||||
expect(padding.length).toBeGreaterThanOrEqual(SSE_PADDING_SIZE);
|
||||
cleanup.dispose();
|
||||
});
|
||||
|
||||
it('sends no padding without a tunnel', () => {
|
||||
const { manager, cleanup } = makeManager();
|
||||
const client = fakeClient();
|
||||
manager.addClient(client.reply, null, false);
|
||||
|
||||
manager.cleanupDeadClients();
|
||||
|
||||
expect(client.writes[0].length).toBeLessThan(200);
|
||||
cleanup.dispose();
|
||||
});
|
||||
|
||||
it('still evicts dead clients instead of heartbeating them', () => {
|
||||
const { manager, cleanup } = makeManager();
|
||||
const alive = fakeClient();
|
||||
const destroyed = fakeClient({ destroyed: true });
|
||||
const unwritable = fakeClient({ writable: false });
|
||||
const exploding = fakeClient({ throwOnAccess: true });
|
||||
for (const c of [alive, destroyed, unwritable, exploding]) manager.addClient(c.reply, null, false);
|
||||
expect(manager.clientCount).toBe(4);
|
||||
|
||||
manager.cleanupDeadClients();
|
||||
|
||||
expect(manager.clientCount).toBe(1);
|
||||
expect(alive.writes).toHaveLength(1);
|
||||
for (const c of [destroyed, unwritable, exploding]) expect(c.writes).toHaveLength(0);
|
||||
cleanup.dispose();
|
||||
});
|
||||
|
||||
it('heartbeats every client on each pass', () => {
|
||||
const { manager, cleanup } = makeManager();
|
||||
const a = fakeClient();
|
||||
const b = fakeClient();
|
||||
manager.addClient(a.reply, null, false);
|
||||
manager.addClient(b.reply, null, false);
|
||||
|
||||
manager.cleanupDeadClients();
|
||||
manager.cleanupDeadClients();
|
||||
|
||||
// The frame carries no session data, so it needs no owner routing and is
|
||||
// written per-client rather than through the scoped broadcast() path.
|
||||
expect(a.writes).toHaveLength(2);
|
||||
expect(b.writes).toHaveLength(2);
|
||||
cleanup.dispose();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* SSE staleness policy.
|
||||
*
|
||||
* `CodemanSseStale.compute(input)` is the pure decision behind app.js's
|
||||
* watchdog: given when the last SSE frame arrived, the transport status and
|
||||
* the browser's online flag, it says whether the stream has gone quiet while
|
||||
* still claiming to be connected: a zombie that has to be rebuilt.
|
||||
*
|
||||
* The regression it guards: the server's liveness keepalive used to be an SSE
|
||||
* `:keepalive` COMMENT, and comments are invisible to `EventSource` by spec.
|
||||
* A stream that stopped delivering without erroring (a proxy that idle-closed
|
||||
* it, a laptop resumed from sleep, a tailnet reconnect) never fired `onerror`,
|
||||
* so the header dot stayed green and tab status dots, sessions created on
|
||||
* another device, and renames all froze until the user reloaded the page.
|
||||
*
|
||||
* Loaded in a plain node VM context (no jsdom), mirroring
|
||||
* test/connection-loss-ui.test.ts.
|
||||
*/
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { resolve } from 'node:path';
|
||||
import vm from 'node:vm';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
type StaleInput = {
|
||||
lastMessageAt?: number | null;
|
||||
now?: number;
|
||||
status?: 'connected' | 'connecting' | 'reconnecting' | 'disconnected' | 'offline';
|
||||
isOnline?: boolean;
|
||||
timeoutMs?: number;
|
||||
};
|
||||
|
||||
function loadPolicy() {
|
||||
const context = vm.createContext({ window: {}, globalThis: {} });
|
||||
const source = readFileSync(resolve(import.meta.dirname, '../src/web/public/constants.js'), 'utf8');
|
||||
vm.runInContext(source, context, { filename: 'constants.js' });
|
||||
return (
|
||||
context.window as {
|
||||
CodemanSseStale: { compute: (input: StaleInput) => boolean; TIMEOUT_MS: number };
|
||||
}
|
||||
).CodemanSseStale;
|
||||
}
|
||||
|
||||
const T0 = 1_000_000;
|
||||
|
||||
describe('SSE staleness policy', () => {
|
||||
it('defaults to three missed 15s heartbeats', () => {
|
||||
const { TIMEOUT_MS } = loadPolicy();
|
||||
expect(TIMEOUT_MS).toBe(45000);
|
||||
});
|
||||
|
||||
it('is not stale while frames keep arriving', () => {
|
||||
const { compute, TIMEOUT_MS } = loadPolicy();
|
||||
expect(compute({ lastMessageAt: T0, now: T0 + TIMEOUT_MS - 1, status: 'connected' })).toBe(false);
|
||||
});
|
||||
|
||||
it('is stale once the threshold is reached', () => {
|
||||
const { compute, TIMEOUT_MS } = loadPolicy();
|
||||
// Boundary is inclusive: exactly three missed heartbeats already means the
|
||||
// stream has been silent through a window it was contractually filling.
|
||||
expect(compute({ lastMessageAt: T0, now: T0 + TIMEOUT_MS, status: 'connected' })).toBe(true);
|
||||
expect(compute({ lastMessageAt: T0, now: T0 + TIMEOUT_MS * 10, status: 'connected' })).toBe(true);
|
||||
});
|
||||
|
||||
it('honours a custom timeoutMs (what a browser test shrinks)', () => {
|
||||
const { compute } = loadPolicy();
|
||||
expect(compute({ lastMessageAt: T0, now: T0 + 999, status: 'connected', timeoutMs: 1000 })).toBe(false);
|
||||
expect(compute({ lastMessageAt: T0, now: T0 + 1000, status: 'connected', timeoutMs: 1000 })).toBe(true);
|
||||
});
|
||||
|
||||
it('is never stale while the transport is already reconnecting', () => {
|
||||
const { compute, TIMEOUT_MS } = loadPolicy();
|
||||
// These states already have the backoff machinery running; firing on top
|
||||
// of them would stack reconnects. This guard is also the loop breaker:
|
||||
// a forced reconnect leaves 'connected' immediately, so the watchdog
|
||||
// cannot re-fire while one is in flight.
|
||||
for (const status of ['connecting', 'reconnecting', 'disconnected', 'offline'] as const) {
|
||||
expect(compute({ lastMessageAt: T0, now: T0 + TIMEOUT_MS * 10, status })).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it('is never stale while the device is offline', () => {
|
||||
const { compute, TIMEOUT_MS } = loadPolicy();
|
||||
// Nothing to reconnect to yet; the connection-loss UI already owns this.
|
||||
expect(compute({ lastMessageAt: T0, now: T0 + TIMEOUT_MS * 10, status: 'connected', isOnline: false })).toBe(false);
|
||||
});
|
||||
|
||||
it('is not stale before any frame has ever arrived', () => {
|
||||
const { compute, TIMEOUT_MS } = loadPolicy();
|
||||
// The clock starts at onopen, and `init` lands immediately after. A zero
|
||||
// stamp means the stream has not opened yet, not that it went quiet. The
|
||||
// constructor optimistically seeds status 'connected' before the first
|
||||
// connect, so without this guard the watchdog would fire on page load.
|
||||
for (const lastMessageAt of [0, null, undefined]) {
|
||||
expect(compute({ lastMessageAt, now: T0 + TIMEOUT_MS * 10, status: 'connected' })).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it('tolerates a missing input object', () => {
|
||||
const { compute } = loadPolicy();
|
||||
expect(compute(undefined as unknown as StaleInput)).toBe(false);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user