Files
Codeman/src/web/routes/reboot-restore-routes.ts
T
Michael GrundbergandClaude Opus 5 da933d70be feat(sessions): offer to rebuild the sessions a host reboot destroyed
A host reboot takes the tmux server down with it, so every pane dies,
reconciliation finds nothing to attach to, and the board comes up empty.
Picking yesterday's work back up meant finding each conversation in history
and resuming it by hand, one at a time.

The boot pass now works out what the reboot killed and leaves it on offer.
It runs inside restoreMuxSessions(), in the window where reconciliation has
reported the dead sessions and cleanupStaleSessions() has not pruned their
records yet, which is the only place the records can still be read. The
board shows a banner, and nothing is created until the user clicks it.

A click rather than an automatic restore is what makes the reboot heuristic
acceptable. The heuristic cannot tell a reboot from a crash that took tmux
down inside the same window, so it decides whether to ASK, never whether to
act: a wrong yes costs a line of text the user dismisses instead of N CLI
processes nobody asked for.

Four things are re-checked when the click arrives rather than trusted from
boot, because hours can pass and the board moves on. The owner's privilege
grant re-resolves through the env clamp. The workspace must still be on
disk. A conversation the user already resumed by hand from the Resume list
is skipped, since two panes running --resume on one conversation would
fight over the same transcript. Entries leave the plan synchronously before
the first await, and the route is single-flighted, so a double-click or two
devices cannot both reach the same entry.

A restored session comes back attached, idle and disarmed. Respawn
controllers and Ralph loops are deliberately not re-armed: a machine that
just came up is the worst moment to turn an autonomous run loose. Its
workspace hooks are installed by the restore route itself, because the
boot-time sweep sits behind a gate that is false after a reboot and has
finished long before the click; without them a session goes silently blind,
with no stop or idle events for respawn, no Approvals Inbox item and no red
tab on a blocking dialog. Stats collection starts the same way.

The pane is new, so the conversation continues and the terminal scrollback
does not. The banner says so rather than letting an empty pane read as a
broken restore.

The plan lives in memory only. A server restart drops it, which costs the
convenience this adds and never the conversation: the conversation is the
transcript under ~/.claude/projects, which the Welcome screen's Resume list
and the Session Manager already read, so a dropped plan returns the user to
resuming by hand.

clampEnvOverridesForOwner moves to src/session-env-clamp.ts, since the
question it answers is about session privilege rather than about HTTP and
it now has a caller outside the route layer. Its test hook stays re-exported
from session-routes.ts.

Claude sessions only for this pass. The other CLIs name their thread in
their own config object, which this does not thread through yet. Remote and
docker sessions are skipped on purpose, because both need another host or a
container to be up and a freshly booted machine cannot promise either.

Refs #411

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:05:55 +02:00

195 lines
9.3 KiB
TypeScript

/**
* @fileoverview Reboot-restore routes: offer back the sessions a host reboot destroyed.
*
* The boot pass leaves a plan in `web/reboot-restore-registry` when the machine
* plausibly rebooted. The board reads it, shows a banner, and the user decides:
* - `GET /api/reboot-restore`: what is on offer, ownership-scoped
* - `POST /api/reboot-restore/restore`: rebuild some or all of it
* - `POST /api/reboot-restore/dismiss`: drop the offer
*
* A click, not the heuristic, is what creates panes. The heuristic only decides
* whether the banner appears, so a wrong yes costs a line of text the user
* dismisses rather than N CLI processes nobody asked for.
*
* Rebuilding is take-then-build: entries leave the plan synchronously at the top
* of the route, before the first `await`, and the whole route is single-flighted,
* so a double-click or two devices cannot put two panes on one conversation.
* Three things are re-checked at click time rather than trusted from boot: the
* owner's privilege grant, the workspace still being on disk, and the
* conversation not already being live because the user resumed it by hand.
*
* A rebuilt session comes back attached, idle and disarmed. Respawn controllers
* and Ralph loops are deliberately not re-armed, and its terminal scrollback is
* gone, because the pane is new. The banner says so.
*/
import { FastifyInstance } from 'fastify';
import { existsSync } from 'node:fs';
import { ApiErrorCode, createErrorResponse, getErrorMessage } from '../../types.js';
import { RebootRestoreRequestSchema } from '../schemas.js';
import { parseBody, getAuthUser, canAccessOwned } from '../route-helpers.js';
import { rebootRestoreRegistry } from '../reboot-restore-registry.js';
import { rejectAlreadyLive, type RebootRestoreEntry, type RebootRestoreRejection } from '../../reboot-restore.js';
import { clampEnvOverridesForOwner } from '../../session-env-clamp.js';
import { Session } from '../../session.js';
import { resolveClaudeModeForUsername } from '../../user-store.js';
import { getCli } from '../../config/cli-registry/registry.js';
import { applyWorkspaceHooks } from '../../hooks-config.js';
import { getLifecycleLog } from '../../session-lifecycle-log.js';
import { STATS_COLLECTION_INTERVAL_MS } from '../../config/server-timing.js';
import { SseEvent } from '../sse-events.js';
import type { SessionAttachmentHistoryItem } from '../../types.js';
import type { SessionPort, EventPort, ConfigPort, InfraPort } from '../ports/index.js';
type RebootRestoreCtx = SessionPort & EventPort & ConfigPort & InfraPort;
/** The banner's view of one restorable session. The record itself never leaves the server. */
function toBannerItem(entry: RebootRestoreEntry) {
return {
id: entry.sessionId,
name: entry.name,
workingDir: entry.workingDir,
mode: entry.mode,
owner: entry.owner,
};
}
export function registerRebootRestoreRoutes(app: FastifyInstance, ctx: RebootRestoreCtx): void {
const accessorFor = (req: Parameters<typeof getAuthUser>[0]) => {
const user = getAuthUser(req);
return (owner: string | undefined) => canAccessOwned(user, owner);
};
// ========== What is on offer ==========
app.get('/api/reboot-restore', async (req) => {
const entries = rebootRestoreRegistry.list(accessorFor(req));
return {
sessions: entries.map(toBannerItem),
// Said plainly here so the banner never implies a full restore: the pane is
// new, so the conversation continues and the terminal history does not.
scrollbackRestored: false,
};
});
// ========== Spend it ==========
app.post('/api/reboot-restore/restore', async (req, reply) => {
const body = parseBody(RebootRestoreRequestSchema, req.body, 'Invalid reboot restore request');
const canAccess = accessorFor(req);
// Take BEFORE the first await: a second click must find nothing to spend.
if (!rebootRestoreRegistry.beginSpending()) {
return reply.code(409).send(createErrorResponse(ApiErrorCode.CONFLICT, 'A reboot restore is already running'));
}
const taken = rebootRestoreRegistry.take(canAccess, body.sessionIds);
try {
if (taken.length === 0) return { restored: [], skipped: [] };
// The plan was built at boot and the board has moved on since. A conversation
// the user resumed by hand from the Resume list is already on screen, and a
// second pane on it would fight the first for the same transcript.
const liveSessionIds = new Set(ctx.sessions.keys());
const liveConversationIds = new Set(
[...ctx.sessions.values()].map((session) => session.claudeSessionId).filter((id): id is string => !!id)
);
const { restore, skipped } = rejectAlreadyLive(taken, liveSessionIds, liveConversationIds);
// An entry nothing rebuilt stays on offer rather than disappearing silently.
rebootRestoreRegistry.restore(skipped.map((s) => taken.find((e) => e.sessionId === s.sessionId)!));
const restored: ReturnType<typeof toBannerItem>[] = [];
const failures: RebootRestoreRejection[] = [...skipped];
const workspaceHooksEnabled = await ctx.getWorkspaceHooksEnabled();
for (const entry of restore) {
// A repo can be deleted between the boot that planned this and the click.
if (!existsSync(entry.workingDir)) {
failures.push({ sessionId: entry.sessionId, reason: 'workspace-missing' });
continue;
}
try {
const saved = entry.state;
const claudeModeConfig = await ctx.getClaudeModeConfig();
const session = new Session({
// The old id is reused on purpose: a pinned record, subagent parents,
// window states and the lifecycle log all key off it, and the unpinned
// record is gone, so there is nothing to collide with.
id: saved.id,
workingDir: saved.workingDir,
mode: saved.mode,
name: saved.name,
createdAt: saved.createdAt,
mux: ctx.mux,
useMux: true,
// No `muxSession`: the reboot took the pane with it, so `startInteractive()`
// takes its create branch and makes a fresh one.
claudeMode: await resolveClaudeModeForUsername(claudeModeConfig.claudeMode, saved.owner),
allowedTools: claudeModeConfig.allowedTools,
resumeSessionId: entry.resumeConversationId,
// Re-resolved against the owner's CURRENT grant, never replayed from the
// record: a grant held when the record was written may be gone now.
envOverrides: await clampEnvOverridesForOwner(
saved.owner,
(saved as { __envOverrides?: Record<string, string> }).__envOverrides
),
effort: saved.effort,
attachmentHistory:
(saved as { __attachmentHistory?: SessionAttachmentHistoryItem[] }).__attachmentHistory ??
saved.attachmentHistory,
lastSubmitAt: saved.lastSubmitAt,
claudeSessionChain: saved.claudeSessionChain,
lastActivityAt: saved.lastActivityAt,
owner: saved.owner,
parentSessionId: saved.parentSessionId,
});
await ctx.addSession(session);
ctx.persistSessionState(session);
await ctx.setupSessionListeners(session);
await session.startInteractive();
// A session without its workspace hooks goes silently blind: no stop or
// idle events for respawn, no Approvals Inbox item, no red tab on a
// blocking dialog. The boot-time sweep finished hours ago, so the click
// path installs them itself. `hooks: 'always'` is the capability that says
// this CLI installs Codeman's hooks into the workspace.
if (workspaceHooksEnabled && getCli(session.mode)?.capabilities.hooks === 'always') {
await applyWorkspaceHooks(session.workingDir, true).catch((err: unknown) =>
console.warn(`[reboot-restore] hook install failed for ${session.workingDir}: ${getErrorMessage(err)}`)
);
}
getLifecycleLog().log({ event: 'recovered', sessionId: session.id, name: session.name });
// Every other open tab and phone needs this; the clicking tab already has
// the response, and the client's handler is an idempotent upsert.
ctx.broadcast(SseEvent.SessionCreated, ctx.getSessionStateWithRespawn(session));
restored.push(toBannerItem(entry));
} catch (err) {
// One workspace that has gone missing must not stop the rest of the pass.
console.error(`[reboot-restore] failed to rebuild ${entry.sessionId}:`, err);
failures.push({ sessionId: entry.sessionId, reason: 'workspace-missing' });
}
}
if (restored.length > 0) {
// A reboot leaves recovery with nothing alive to find, so its own block never
// started the stats collector. This clears and re-arms its interval, so it is
// safe to call whether or not the collector is already running.
ctx.mux.startStatsCollection(STATS_COLLECTION_INTERVAL_MS);
}
return { restored, skipped: failures };
} finally {
rebootRestoreRegistry.endSpending();
}
});
// ========== Drop it ==========
app.post('/api/reboot-restore/dismiss', async (req) => {
const dismissed = rebootRestoreRegistry.clear(accessorFor(req));
return { dismissed };
});
}