feat(sessions): offer to rebuild the sessions a host reboot destroyed

A host reboot takes the tmux server down with it, so every pane dies,
reconciliation finds nothing to attach to, and the board comes up empty.
Picking yesterday's work back up meant finding each conversation in history
and resuming it by hand, one at a time.

The boot pass now works out what the reboot killed and leaves it on offer.
It runs inside restoreMuxSessions(), in the window where reconciliation has
reported the dead sessions and cleanupStaleSessions() has not pruned their
records yet, which is the only place the records can still be read. The
board shows a banner, and nothing is created until the user clicks it.

A click rather than an automatic restore is what makes the reboot heuristic
acceptable. The heuristic cannot tell a reboot from a crash that took tmux
down inside the same window, so it decides whether to ASK, never whether to
act: a wrong yes costs a line of text the user dismisses instead of N CLI
processes nobody asked for.

Four things are re-checked when the click arrives rather than trusted from
boot, because hours can pass and the board moves on. The owner's privilege
grant re-resolves through the env clamp. The workspace must still be on
disk. A conversation the user already resumed by hand from the Resume list
is skipped, since two panes running --resume on one conversation would
fight over the same transcript. Entries leave the plan synchronously before
the first await, and the route is single-flighted, so a double-click or two
devices cannot both reach the same entry.

A restored session comes back attached, idle and disarmed. Respawn
controllers and Ralph loops are deliberately not re-armed: a machine that
just came up is the worst moment to turn an autonomous run loose. Its
workspace hooks are installed by the restore route itself, because the
boot-time sweep sits behind a gate that is false after a reboot and has
finished long before the click; without them a session goes silently blind,
with no stop or idle events for respawn, no Approvals Inbox item and no red
tab on a blocking dialog. Stats collection starts the same way.

The pane is new, so the conversation continues and the terminal scrollback
does not. The banner says so rather than letting an empty pane read as a
broken restore.

The plan lives in memory only. A server restart drops it, which costs the
convenience this adds and never the conversation: the conversation is the
transcript under ~/.claude/projects, which the Welcome screen's Resume list
and the Session Manager already read, so a dropped plan returns the user to
resuming by hand.

clampEnvOverridesForOwner moves to src/session-env-clamp.ts, since the
question it answers is about session privilege rather than about HTTP and
it now has a caller outside the route layer. Its test hook stays re-exported
from session-routes.ts.

Claude sessions only for this pass. The other CLIs name their thread in
their own config object, which this does not thread through yet. Remote and
docker sessions are skipped on purpose, because both need another host or a
container to be up and a freshly booted machine cannot promise either.

Refs #411

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michael Grundberg
2026-09-16 08:05:55 +02:00
co-authored by Claude Opus 5
parent bd286bf502
commit da933d70be
15 changed files with 1462 additions and 67 deletions
+367
View File
@@ -0,0 +1,367 @@
/**
* @fileoverview The decision half of reboot restore, and proof that the existing
* recovery construction path can CREATE a resumed pane.
*
* Three things are under test. `src/reboot-restore.ts` decides whether the
* machine rebooted and which dead sessions may be offered back. The plan
* registry in `src/web/reboot-restore-registry.ts` holds that offer between the
* boot that builds it and the click that spends it. The third is the claim the
* whole feature rests on: a `Session` built the way `restoreMuxSessions()`
* already builds one, but given no `muxSession` and a `resumeSessionId`, creates
* a fresh pane that resumes the old conversation. If that holds, the restore
* needs no new session-creation service.
*
* `reconcileSessions()` reports every session ALIVE under vitest, so the
* server's own boot pass cannot be reached from here. The decision logic is
* therefore driven directly, and the construction claim is driven through a real
* `Session` against the in-memory tmux layer vitest substitutes.
*/
import { mkdirSync, rmSync } from 'node:fs';
import { homedir } from 'node:os';
import { join } from 'node:path';
import { afterEach, describe, expect, it } from 'vitest';
import { Session } from '../src/session.js';
import { TmuxManager } from '../src/tmux-manager.js';
import type { SessionState } from '../src/types.js';
import {
looksLikeHostReboot,
newestPersistedActivity,
planRebootRestore,
rejectAlreadyLive,
resolveResumeConversationId,
type RebootRestoreEntry,
} from '../src/reboot-restore.js';
import { RebootRestoreRegistry } from '../src/web/reboot-restore-registry.js';
const HOUR = 60 * 60 * 1000;
const NOW = 1_760_000_000_000;
function persistedSession(overrides: Partial<SessionState> & { id: string }): SessionState {
return {
pid: 99999,
status: 'idle',
workingDir: '/tmp/spike',
currentTaskId: null,
createdAt: NOW - 4 * HOUR,
lastActivityAt: NOW - 2 * HOUR,
mode: 'claude',
...overrides,
} as SessionState;
}
describe('reboot detection', () => {
const base = {
livePaneCount: 0,
deadSessionCount: 2,
// The host came up 10 minutes ago, well after the sessions were last active.
uptimeSeconds: 600,
newestPersistedActivityAt: NOW - 2 * HOUR,
now: NOW,
};
it('calls it a reboot when the socket is empty and the host booted after the last activity', () => {
expect(looksLikeHostReboot(base)).toBe(true);
});
it('refuses when some panes survived, which is an ordinary server restart', () => {
expect(looksLikeHostReboot({ ...base, livePaneCount: 3 })).toBe(false);
});
it('refuses on a long-uptime host, where someone wiped the tmux socket by hand', () => {
// Up for 30 days: the sessions were active long AFTER this boot, so the panes
// went away for some reason other than the machine restarting.
expect(looksLikeHostReboot({ ...base, uptimeSeconds: 30 * 24 * 60 * 60 })).toBe(false);
});
it('refuses when nothing died', () => {
expect(looksLikeHostReboot({ ...base, deadSessionCount: 0 })).toBe(false);
});
it('reads the newest activity stamp across the persisted records', () => {
const persisted = {
a: persistedSession({ id: 'a', lastActivityAt: NOW - 5 * HOUR }),
b: persistedSession({ id: 'b', lastActivityAt: NOW - 1 * HOUR }),
};
expect(newestPersistedActivity(persisted)).toBe(NOW - 1 * HOUR);
});
});
describe('which dead sessions may be rebuilt', () => {
it('rebuilds a session that was simply running when the power went out', () => {
const persisted = { live: persistedSession({ id: 'live', status: 'busy' }) };
const plan = planRebootRestore(['live'], persisted, () => true);
expect(plan.restore.map((s) => s.sessionId)).toEqual(['live']);
});
it('never revives a session the user killed while pinned (COD-142 demotes it to stopped)', () => {
const persisted = { killed: persistedSession({ id: 'killed', status: 'stopped', pinned: true }) };
const plan = planRebootRestore(['killed'], persisted, () => true);
expect(plan.restore).toEqual([]);
expect(plan.skipped).toEqual([{ sessionId: 'killed', reason: 'intentionally-ended' }]);
});
it('never revives a session whose record an unpinned kill already deleted', () => {
const plan = planRebootRestore(['gone'], {}, () => true);
expect(plan.restore).toEqual([]);
expect(plan.skipped).toEqual([{ sessionId: 'gone', reason: 'no-persisted-record' }]);
});
it('never revives a pane whose PTY-exit breaker had tripped', () => {
const persisted = { crashy: persistedSession({ id: 'crashy', respawnBlocked: true }) };
expect(planRebootRestore(['crashy'], persisted, () => true).skipped[0].reason).toBe('respawn-blocked');
});
it('leaves remote sessions to the COD-108 reconnect watcher', () => {
const persisted = {
r: persistedSession({
id: 'r',
remote: { hostId: 'h', host: 'example.test', username: 'u', sessionName: 'n', owned: true },
} as Partial<SessionState> & { id: string }),
};
expect(planRebootRestore(['r'], persisted, () => true).skipped[0].reason).toBe('remote-or-docker');
});
it('leaves docker sessions alone, since the container may not be up', () => {
const persisted = {
d: persistedSession({ id: 'd', docker: { containerId: 'abc', caseId: 'c' } } as Partial<SessionState> & {
id: string;
}),
};
expect(planRebootRestore(['d'], persisted, () => true).skipped[0].reason).toBe('remote-or-docker');
});
it('skips a CLI whose history the claude transcript reader does not understand', () => {
const persisted = { c: persistedSession({ id: 'c', mode: 'codex' }) };
expect(planRebootRestore(['c'], persisted, () => true).skipped[0].reason).toBe('unsupported-mode');
});
});
describe('a workspace that is no longer on disk', () => {
it('is kept out of the offer, so a click cannot scaffold a deleted repo', () => {
const persisted = { gone: persistedSession({ id: 'gone', workingDir: '/tmp/deleted-repo' }) };
const plan = planRebootRestore(['gone'], persisted, () => false);
expect(plan.restore).toEqual([]);
expect(plan.skipped).toEqual([{ sessionId: 'gone', reason: 'workspace-missing' }]);
});
it('is judged per session, not for the batch', () => {
const persisted = {
kept: persistedSession({ id: 'kept', workingDir: '/tmp/still-here' }),
gone: persistedSession({ id: 'gone', workingDir: '/tmp/deleted-repo' }),
};
const plan = planRebootRestore(['kept', 'gone'], persisted, (dir) => dir === '/tmp/still-here');
expect(plan.restore.map((entry) => entry.sessionId)).toEqual(['kept']);
expect(plan.skipped.map((s) => s.reason)).toEqual(['workspace-missing']);
});
});
describe('a conversation that came back on its own before the click', () => {
const entry: RebootRestoreEntry = {
sessionId: 'abc',
workingDir: '/tmp/spike',
mode: 'claude',
resumeConversationId: 'conv-1',
state: persistedSession({ id: 'abc' }),
};
it('is skipped when the user resumed it by hand from the Resume list', () => {
// Same conversation, different session id: the Resume list creates a NEW id.
const result = rejectAlreadyLive([entry], new Set(['other']), new Set(['conv-1']));
expect(result.restore).toEqual([]);
expect(result.skipped).toEqual([{ sessionId: 'abc', reason: 'already-live' }]);
});
it('is skipped when a session with that id is already on the board', () => {
const result = rejectAlreadyLive([entry], new Set(['abc']), new Set());
expect(result.skipped).toEqual([{ sessionId: 'abc', reason: 'already-live' }]);
});
it('is rebuilt when neither its id nor its conversation is live', () => {
const result = rejectAlreadyLive([entry], new Set(['other']), new Set(['conv-other']));
expect(result.restore.map((e) => e.sessionId)).toEqual(['abc']);
expect(result.skipped).toEqual([]);
});
});
describe('the plan the banner spends', () => {
const all = () => true;
const entryFor = (sessionId: string, owner?: string): RebootRestoreEntry => ({
sessionId,
owner,
workingDir: '/tmp/spike',
mode: 'claude',
resumeConversationId: `conv-${sessionId}`,
state: persistedSession({ id: sessionId, owner }),
});
it('hands an entry to the first caller and nothing to the second', () => {
const registry = new RebootRestoreRegistry();
registry.set([entryFor('a'), entryFor('b')]);
expect(registry.take(all).map((e) => e.sessionId)).toEqual(['a', 'b']);
// The double-click: two panes on one conversation is what this prevents.
expect(registry.take(all)).toEqual([]);
});
it('spends only the ids a caller asked for', () => {
const registry = new RebootRestoreRegistry();
registry.set([entryFor('a'), entryFor('b')]);
expect(registry.take(all, ['b']).map((e) => e.sessionId)).toEqual(['b']);
expect(registry.list(all).map((e) => e.sessionId)).toEqual(['a']);
});
it("shows a user their own sessions and leaves another owner's alone", () => {
const registry = new RebootRestoreRegistry();
registry.set([entryFor('mine', 'alice'), entryFor('theirs', 'bob')]);
const asAlice = (owner: string | undefined) => owner === 'alice';
expect(registry.list(asAlice).map((e) => e.sessionId)).toEqual(['mine']);
expect(registry.take(asAlice).map((e) => e.sessionId)).toEqual(['mine']);
// Bob's entry is still on offer for Bob.
expect(registry.list(() => true).map((e) => e.sessionId)).toEqual(['theirs']);
});
it('puts back an entry that no pane was created for', () => {
const registry = new RebootRestoreRegistry();
registry.set([entryFor('a')]);
const taken = registry.take(all);
registry.restore(taken);
expect(registry.list(all).map((e) => e.sessionId)).toEqual(['a']);
});
it('runs one restore at a time', () => {
const registry = new RebootRestoreRegistry();
expect(registry.beginSpending()).toBe(true);
expect(registry.beginSpending()).toBe(false);
registry.endSpending();
expect(registry.beginSpending()).toBe(true);
});
it('drops what a dismiss cleared', () => {
const registry = new RebootRestoreRegistry();
registry.set([entryFor('a'), entryFor('b')]);
expect(registry.clear(all)).toBe(2);
expect(registry.list(all)).toEqual([]);
});
it('forgets a plan nobody took for a day', () => {
const registry = new RebootRestoreRegistry();
registry.set([entryFor('a')]);
const dayLater = Date.now() + 25 * HOUR;
const realNow = Date.now;
Date.now = () => dayLater;
try {
expect(registry.list(all)).toEqual([]);
} finally {
Date.now = realNow;
}
});
});
describe('which conversation a rebuilt pane resumes', () => {
it('prefers the chain tail, the conversation the CLI reported last', () => {
const state = persistedSession({
id: 'sess-1',
resumeSessionId: 'launch-id',
claudeSessionChain: ['launch-id', 'after-clear'],
});
expect(resolveResumeConversationId(state)).toBe('after-clear');
});
it('falls back to the id the session originally resumed', () => {
const state = persistedSession({ id: 'sess-1', resumeSessionId: 'resumed-id' });
expect(resolveResumeConversationId(state)).toBe('resumed-id');
});
it('falls back to the session id, which is what Claude was launched with', () => {
expect(resolveResumeConversationId(persistedSession({ id: 'sess-1' }))).toBe('sess-1');
});
});
describe('the recovery construction path can create a resumed pane', () => {
const workingDir = join(homedir(), 'codeman-cases', 'reboot-restore-spike');
const sessions: Session[] = [];
afterEach(() => {
for (const s of sessions.splice(0)) s.stop();
rmSync(workingDir, { recursive: true, force: true });
});
/** Built exactly as the reboot pass builds one: no `muxSession`, plus a resume id. */
function rebuildFromPersistedState(state: SessionState, mux: TmuxManager): Session {
mkdirSync(workingDir, { recursive: true });
const session = new Session({
id: state.id,
workingDir,
mode: state.mode,
name: state.name,
createdAt: state.createdAt,
mux,
useMux: true,
resumeSessionId: resolveResumeConversationId(state),
owner: state.owner,
lastActivityAt: state.lastActivityAt,
claudeSessionChain: state.claudeSessionChain,
});
sessions.push(session);
return session;
}
it('creates a NEW mux session rather than needing one to attach to', async () => {
const mux = new TmuxManager();
const state = persistedSession({ id: 'aaaaaaa1-1111-4111-8111-111111111111', name: 'w1-spike' });
const session = rebuildFromPersistedState(state, mux);
expect(mux.getSessions()).toHaveLength(0);
await session.startInteractive();
const created = mux.getSessions();
expect(created).toHaveLength(1);
expect(created[0].sessionId).toBe('aaaaaaa1-1111-4111-8111-111111111111');
expect(created[0].workingDir).toBe(workingDir);
});
it('comes back pointed at the conversation the pane was holding', async () => {
const mux = new TmuxManager();
const state = persistedSession({
id: 'aaaaaaa2-2222-4222-8222-222222222222',
resumeSessionId: 'launch-id',
claudeSessionChain: ['launch-id', 'after-clear'],
});
const session = rebuildFromPersistedState(state, mux);
await session.startInteractive();
// The chain tail wins: a `/clear` before the reboot moved the CLI off the launch id.
expect(session.claudeSessionId).toBe('after-clear');
});
it('comes back idle, with no prompt sent and no autonomous loop armed', async () => {
const mux = new TmuxManager();
const state = persistedSession({
id: 'aaaaaaa3-3333-4333-8333-333333333333',
ralphEnabled: true,
respawnEnabled: true,
});
const session = rebuildFromPersistedState(state, mux);
await session.startInteractive();
// No prompt was queued: nothing is waiting on a task. The status itself is not
// assertable here, because the test PTY echoes and the activity detector reads
// that echo as work; in production the pane settles once the CLI finishes booting.
expect(session.currentTaskId).toBeNull();
// The pass never touches the tracker, so a persisted Ralph loop stays cold.
expect(session.ralphTracker.enabled).toBe(false);
});
it('keeps the owner it was persisted with, there being no request to read one from', async () => {
const mux = new TmuxManager();
const state = persistedSession({ id: 'aaaaaaa4-4444-4444-8444-444444444444', owner: 'alice' });
const session = rebuildFromPersistedState(state, mux);
await session.startInteractive();
expect(session.owner).toBe('alice');
expect(mux.getSessions()[0].owner).toBe('alice');
});
});
+175
View File
@@ -0,0 +1,175 @@
/**
* Reboot-restore route tests (src/web/routes/reboot-restore-routes.ts) via
* app.inject(), no live port.
*
* Every entry these tests put on offer names a workspace that does not exist, so
* the route's click-time workspace check rejects it before any `Session` is
* constructed. That keeps the tests on the route's own guards — taking, scoping,
* single-flighting and re-checking — and leaves pane creation to
* test/reboot-restore.test.ts, which drives a real `Session` for it.
*
* The routes read the process-wide `rebootRestoreRegistry` singleton, so every
* test resets it; a leaked entry would bleed into the next one.
*/
import { describe, it, expect, afterEach } from 'vitest';
import Fastify, { type FastifyInstance } from 'fastify';
import fastifyCookie from '@fastify/cookie';
import { registerRebootRestoreRoutes } from '../../src/web/routes/reboot-restore-routes.js';
import { rebootRestoreRegistry } from '../../src/web/reboot-restore-registry.js';
import { installRouteErrorHandler } from '../../src/web/route-error-handler.js';
import { httpStatusForErrorCode, type ApiErrorCode } from '../../src/types.js';
import { createMockRouteContext } from '../mocks/index.js';
import type { RebootRestoreEntry } from '../../src/reboot-restore.js';
import type { SessionState } from '../../src/types.js';
async function createHarness(authUser?: { username: string; role: 'admin' | 'user' }): Promise<FastifyInstance> {
const app = Fastify({ logger: false });
await app.register(fastifyCookie);
if (authUser) {
app.addHook('onRequest', async (req) => {
(req as unknown as { authUser: typeof authUser }).authUser = authUser;
});
}
registerRebootRestoreRoutes(app, createMockRouteContext() as never);
app.addHook('preSerialization', (req, reply, payload: unknown, done) => {
if (!req.url.startsWith('/api')) return done(null, payload);
if (payload === null || typeof payload !== 'object') return done(null, payload);
const p = payload as { success?: unknown; errorCode?: unknown };
if (p.success === false) {
if (reply.statusCode === 200 && typeof p.errorCode === 'string') {
reply.code(httpStatusForErrorCode(p.errorCode as ApiErrorCode));
}
return done(null, payload);
}
if (p.success === true) return done(null, payload);
return done(null, { success: true, data: payload });
});
installRouteErrorHandler(app);
await app.ready();
return app;
}
/** An entry whose workspace is deliberately absent, so no pane is ever created. */
function offerEntry(sessionId: string, owner?: string): RebootRestoreEntry {
return {
sessionId,
name: `session ${sessionId}`,
workingDir: `/tmp/codeman-reboot-restore-missing/${sessionId}`,
owner,
mode: 'claude',
resumeConversationId: `conv-${sessionId}`,
state: {
id: sessionId,
pid: null,
status: 'idle',
workingDir: `/tmp/codeman-reboot-restore-missing/${sessionId}`,
currentTaskId: null,
createdAt: 1_760_000_000_000,
mode: 'claude',
owner,
} as SessionState,
};
}
afterEach(() => {
rebootRestoreRegistry.reset();
});
describe('GET /api/reboot-restore', () => {
it('reports nothing when no reboot left anything behind', async () => {
const app = await createHarness();
const res = await app.inject({ method: 'GET', url: '/api/reboot-restore' });
expect(res.statusCode).toBe(200);
expect(res.json().data.sessions).toEqual([]);
await app.close();
});
it('names what is on offer, and says the scrollback is not coming back', async () => {
rebootRestoreRegistry.set([offerEntry('a'), offerEntry('b')]);
const app = await createHarness();
const body = (await app.inject({ method: 'GET', url: '/api/reboot-restore' })).json().data;
expect(body.sessions.map((s: { id: string }) => s.id)).toEqual(['a', 'b']);
expect(body.scrollbackRestored).toBe(false);
await app.close();
});
it('never carries the persisted record itself to the browser', async () => {
rebootRestoreRegistry.set([offerEntry('a', 'alice')]);
const app = await createHarness({ username: 'alice', role: 'admin' });
const body = (await app.inject({ method: 'GET', url: '/api/reboot-restore' })).json().data;
expect(Object.keys(body.sessions[0]).sort()).toEqual(['id', 'mode', 'name', 'owner', 'workingDir']);
expect(body.sessions[0].state).toBeUndefined();
await app.close();
});
});
describe('POST /api/reboot-restore/restore', () => {
it('spends the offer, so a second click finds nothing left to spend', async () => {
rebootRestoreRegistry.set([offerEntry('a')]);
const app = await createHarness();
const first = (await app.inject({ method: 'POST', url: '/api/reboot-restore/restore', payload: {} })).json().data;
// The workspace is gone, so nothing was rebuilt — but the entry was taken.
expect(first.restored).toEqual([]);
expect(first.skipped).toEqual([{ sessionId: 'a', reason: 'workspace-missing' }]);
const second = (await app.inject({ method: 'POST', url: '/api/reboot-restore/restore', payload: {} })).json().data;
expect(second.restored).toEqual([]);
expect(second.skipped).toEqual([]);
await app.close();
});
it('spends only the sessions the click named', async () => {
rebootRestoreRegistry.set([offerEntry('a'), offerEntry('b')]);
const app = await createHarness();
const res = await app.inject({
method: 'POST',
url: '/api/reboot-restore/restore',
payload: { sessionIds: ['b'] },
});
expect(res.json().data.skipped).toEqual([{ sessionId: 'b', reason: 'workspace-missing' }]);
const left = (await app.inject({ method: 'GET', url: '/api/reboot-restore' })).json().data;
expect(left.sessions.map((s: { id: string }) => s.id)).toEqual(['a']);
await app.close();
});
it('refuses a body it does not recognise rather than guessing', async () => {
const app = await createHarness();
const res = await app.inject({
method: 'POST',
url: '/api/reboot-restore/restore',
payload: { sessionIds: 'not-an-array' },
});
expect(res.statusCode).toBeGreaterThanOrEqual(400);
await app.close();
});
it('turns a second concurrent restore away rather than interleaving it', async () => {
rebootRestoreRegistry.set([offerEntry('a')]);
// Claimed by a restore already in flight.
expect(rebootRestoreRegistry.beginSpending()).toBe(true);
const app = await createHarness();
const res = await app.inject({ method: 'POST', url: '/api/reboot-restore/restore', payload: {} });
expect(res.statusCode).toBe(409);
rebootRestoreRegistry.endSpending();
await app.close();
});
});
describe('POST /api/reboot-restore/dismiss', () => {
it('drops the offer and leaves the banner with nothing to show', async () => {
rebootRestoreRegistry.set([offerEntry('a'), offerEntry('b')]);
const app = await createHarness();
const res = await app.inject({ method: 'POST', url: '/api/reboot-restore/dismiss', payload: {} });
expect(res.json().data.dismissed).toBe(2);
const after = (await app.inject({ method: 'GET', url: '/api/reboot-restore' })).json().data;
expect(after.sessions).toEqual([]);
await app.close();
});
});