feat(remote): wake a sleeping host when a session is created or attached

Pressing Run on a remote case whose host was asleep failed with `could not verify tmux on remote host 192.168.50.137: …` — an ssh error that blames tmux for a machine that is merely suspended. The only wake paths were typed input on an established session and the banner's Wake button, so OPENING a session (the moment the user actually decides to use that host) had none. `RemoteWakeRegistry.ensureHostAwake()` reuses the existing probe/wake/readiness machinery for a host that has no session yet, and is wired into the two user-initiated create paths: `POST /api/quick-start` for a remote case (before the tmux prereq probe, which is what surfaced the misleading error) and `POST /api/sessions` with `attachRemoteSession`. A host without a wake target is not even probed, so its behavior and latency are byte-identical. The wake is blocking — the caller gets the session or an error — but bounded by REMOTE_WAKE_REQUEST_READY_TIMEOUT_MS (40 s) instead of the 90 s session default, because the dashboard sits behind a reverse proxy whose default `proxy_read_timeout` is 60 s: a longer wait would be cut off at the proxy while the session was still being created. The budget has to cover the whole request (40 s wake + 1.5 s probe + the tmux probe's own 15 s = 56.5 s worst case), which is why it is 40 s and not 45. A timeout now says the host did not come back, and an unreachable host without a wake target says so instead of pointing at tmux. The wiring is deliberately in the HTTP ROUTE, never in the shared session service: `cron-service.ts` builds sessions there with nobody waiting on the answer, and a wake on that path would power the host on for every schedule — the timer-driven re-wake invariant #1 exists to prevent. Both halves are asserted (importers of `remote-wake`, and `ensureHostAwake` having exactly one caller file), so a future caller has to come through the guard test. A rejection from the wake IO is caught too: a broken target must fail the wake, not the route. `remote:hostWaking`/`remote:hostWakeFailed` now carry `forNewSession` for the session-less case, where "input is queued" would be untrue; the toast then reads "the session starts when it is back". Live wake numbers are unchanged (this reuses the measured ~12 s S3 path); the route behavior is covered by new tests in session-routes.test.ts with an injected registry, so no test opens a real socket or ssh.
2026-10-04 06:29:42 +02:00 · 2026-09-15 22:37:37 +02:00
parent 8dfc965d13
commit d0a5a583cd
7 changed files with 518 additions and 40 deletions
@@ -8,11 +8,16 @@
 * lost with no error anywhere — the failure this module exists to close.
 *
 * Design (deliberately narrow, see docs/remote-sessions.md §Wake-on-LAN):
- *  - ONLY real user input wakes a host. The auto-reconnect watcher and
- *    boot-recovery must never wake one, or a host would be re-woken ~45 s after
- *    each suspend and could never stay asleep (the "keepalive pings a sleeping
- *    host" failure already solved for a different consumer by
- *    `hufflepuff-mcp-lazy`).
+ *  - An EXPLICIT request wakes a host, and nothing else: user input on an
+ *    established session (`handleInput`), the wake button (`ensureAwake`), or the
+ *    user's own session create/attach request (`ensureHostAwake`, wired in the HTTP
+ *    routes). Everything that runs on a TIMER — the auto-reconnect watcher, boot
+ *    recovery, the reachability probe, session discovery — must never wake one, or
+ *    a host would be re-woken ~45 s after each suspend and could never stay asleep
+ *    (the "keepalive pings a sleeping host" failure already solved for a different
+ *    consumer by `hufflepuff-mcp-lazy`). The create path is deliberately wired in
+ *    `session-routes.ts` and NOT in the shared session service, because
+ *    `cron-service.ts` builds sessions there without a user waiting on the answer.
 *  - Detection is a cheap TCP connect to the SSH port (no auth, no ssh client,
 *    a few hundred bytes — below any meaningful activity threshold), throttled
 *    per session. No SSH keepalive is added to the launch command: keepalives
@@ -41,6 +46,17 @@ export const REMOTE_WAKE_PROBE_TIMEOUT_MS = 1_500;
 export const REMOTE_WAKE_READY_INTERVAL_MS = 1_500;
 /** Bounded wait for the host to come back after the wake command ran. */
 export const REMOTE_WAKE_READY_TIMEOUT_MS = 90_000;
+/**
+ * Budget for a wake that an HTTP REQUEST is waiting on (session create/attach).
+ * Deliberately shorter than {@link REMOTE_WAKE_READY_TIMEOUT_MS}: the dashboard is
+ * served through a reverse proxy whose default `proxy_read_timeout` is 60 s, so a
+ * 90 s wait would be cut off AT THE PROXY while the session was still being built —
+ * the browser reports a failure for a session that exists. The budget has to cover
+ * the WHOLE request, not just the wait: 40 s here + the 1.5 s reachability probe +
+ * the tmux prereq probe's own 15 s timeout = 56.5 s worst case, still under 60 s.
+ * A warm S3 resume measures ~12 s, so 40 s is >3× the observed wake.
+ */
+export const REMOTE_WAKE_REQUEST_READY_TIMEOUT_MS = 40_000;
 /** The wake command itself must not hang the wake flow. */
 export const REMOTE_WAKE_COMMAND_TIMEOUT_MS = 10_000;
 /**
@@ -224,7 +240,7 @@ export interface RemoteWakeDeps {
  /** Run the resolved wake target (magic packet or host command). Resolves false on failure. */
  wake(target: NonNullable<WakeTarget>): Promise<boolean>;
  /** Poll until the woken host accepts connections again. */
-  waitUntilReady(remote: WakeableRemote): Promise<boolean>;
+  waitUntilReady(remote: WakeableRemote, opts?: { timeoutMs?: number }): Promise<boolean>;
  /** Sleep helper (injected for tests). */
  delay(ms: number): Promise<void>;
  /** Notify the COD-108 watcher so an exhausted backoff is reset. */
@@ -264,6 +280,25 @@ export function wakeConfigured(remote: WakeableRemote | undefined): WakeConfigur
  return target.kind;
 }

+/**
+ * Outcome of waking a host for a caller that has NO session yet (the create/attach
+ * routes). A union rather than a boolean because the three cases need different
+ * handling: `'no-target'` must leave the caller's behavior byte-identical (no probe,
+ * no extra latency for a host without WoL), and only `'failed'` is an error that
+ * deserves its own message instead of the caller's usual one.
+ */
+export type HostWakeOutcome = 'no-target' | 'ready' | 'failed';
+
+/**
+ * State key for a host-scoped wake. Prefixed so it can never collide with a session
+ * id, and keyed on the HOST rather than the case: two cases on one host share a
+ * single in-flight wake and one probe verdict. Such an entry is tiny (no input
+ * buffer) and bounded by the number of configured hosts, so it is never dropped.
+ */
+function hostWakeKey(hostId: string): string {
+  return `host:${hostId}`;
+}
+
 /** Per-session wake bookkeeping. */
 interface WakeState {
  probedAt: number;
@@ -391,6 +426,67 @@ export class RemoteWakeRegistry {
    return this.wake(session);
  }

+  /**
+   * Host-scoped reachability, for a caller that has no session yet (create/attach).
+   * Shares the per-HOST probe state with {@link ensureHostAwake}, so the probe the
+   * wake flow just paid for also answers "was that ssh failure really a sleeping
+   * machine?". Never wakes anything — it is a question, not an action.
+   */
+  async checkHostReachable(remote: WakeableRemote, opts: { force?: boolean; ttlMs?: number } = {}): Promise<boolean> {
+    const state = this._state(hostWakeKey(remote.hostId));
+    const ttl = opts.force ? 0 : (opts.ttlMs ?? REMOTE_WAKE_REACHABILITY_TTL_MS);
+    if (Date.now() - state.probedAt >= ttl) {
+      state.probedAt = Date.now();
+      state.reachable = await this.deps.probe(remote);
+    }
+    return state.reachable === true;
+  }
+
+  /**
+   * Wake a host for a REQUEST that is waiting on it — the session create/attach
+   * routes, where there is no session to reattach and no input to buffer yet.
+   *
+   * `'no-target'` returns without probing, so a host without WoL config costs
+   * nothing and behaves exactly as before. Single-flight per host, so a double click
+   * (or two cases on the same host) sends one packet and shares one readiness poll.
+   */
+  async ensureHostAwake(remote: WakeableRemote, opts: { timeoutMs?: number } = {}): Promise<HostWakeOutcome> {
+    if (!resolveWakeTarget(remote)) return 'no-target';
+    const state = this._state(hostWakeKey(remote.hostId));
+    if (state.waking) return (await state.waking) ? 'ready' : 'failed';
+
+    state.probedAt = Date.now();
+    state.reachable = await this.deps.probe(remote);
+    if (state.reachable) return 'ready';
+    this.deps.log?.(`[RemoteWake] ${remote.label} (${remote.host}) is unreachable — waking it for a new session`);
+    return (await this.wakeHost(remote, opts)) ? 'ready' : 'failed';
+  }
+
+  /**
+   * Single-flight wake for a host with no session (see {@link ensureHostAwake}).
+   * A write into the same `waking` slot the session flow uses, so the two can never
+   * run two readiness polls against one host from the same key space.
+   */
+  private async wakeHost(remote: WakeableRemote, opts: { timeoutMs?: number }): Promise<boolean> {
+    const state = this._state(hostWakeKey(remote.hostId));
+    if (state.waking) return state.waking;
+    state.waking = (async (): Promise<boolean> => {
+      try {
+        return await this._wakeAndWait(remote, state, { timeoutMs: opts.timeoutMs, forNewSession: true });
+      } catch (err) {
+        // Injected IO is documented not to throw, but a rejected promise here would
+        // surface as an unhandled rejection AND take the route down with it (the
+        // session path catches for exactly this reason). A broken wake target must
+        // fail the wake, never the create route beyond its own error response.
+        this.deps.log?.(`[RemoteWake] unexpected failure: ${err instanceof Error ? err.message : String(err)}`);
+        return false;
+      } finally {
+        state.waking = null;
+      }
+    })();
+    return state.waking;
+  }
+
  /**
   * Single-flight wake: probe-free (the caller already knows the host is down),
   * run the wake command, poll for readiness, reattach the pane, flush the buffer.
@@ -405,29 +501,9 @@ export class RemoteWakeRegistry {
    state.waking = (async (): Promise<boolean> => {
      const id = session.id;
      try {
-        this.deps.broadcast?.('remote:hostWaking', { sessionId: id, hostId: remote.hostId, label: remote.label });
-        this.deps.log?.(`[RemoteWake] waking ${remote.label} (${remote.host}) via ${target.kind} for session ${id}`);
+        const ready = await this._wakeAndWait(remote, state, { sessionId: id });
+        if (!ready) return false;

-        const woke = await this.deps.wake(target);
-        if (!woke) {
-          this.deps.log?.(
-            `[RemoteWake] wake failed for ${remote.label}: ${target.kind === 'command' ? target.command : 'magic packet'}`
-          );
-        }
-
-        const ready = await this.deps.waitUntilReady(remote);
-        if (!ready) {
-          this.deps.log?.(`[RemoteWake] ${remote.label} did not come back — input stays buffered`);
-          this.deps.broadcast?.('remote:hostWakeFailed', { sessionId: id, hostId: remote.hostId, label: remote.label });
-          // Reset the probe state so the NEXT user input probes and retries
-          // instead of trusting a stale "down" verdict forever.
-          state.probedAt = 0;
-          state.reachable = undefined;
-          return false;
-        }
-
-        state.reachable = true;
-        state.probedAt = Date.now();
        const reattached = await session.reattachRemote();
        if (!reattached) {
          this.deps.log?.(`[RemoteWake] ${remote.label} is up but the pane could not be reattached`);
@@ -453,6 +529,57 @@ export class RemoteWakeRegistry {
    return state.waking;
  }

+  /**
+   * Broadcast + run the wake target + wait for SSH. Shared by the session flow (which
+   * then reattaches and flushes the buffer) and the create/attach flow (which has no
+   * pane yet). On failure the probe state is reset so the NEXT attempt probes and
+   * retries instead of trusting a stale "down" verdict forever.
+   */
+  private async _wakeAndWait(
+    remote: WakeableRemote,
+    state: WakeState,
+    opts: { sessionId?: string; timeoutMs?: number; forNewSession?: boolean } = {}
+  ): Promise<boolean> {
+    const target = resolveWakeTarget(remote);
+    if (!target) return true;
+    const forWhat = opts.sessionId ? `for session ${opts.sessionId}` : 'for a new session';
+    // No `sessionId` for a create-path wake: the toast handler is then the only one
+    // that acts (a banner for a session that does not exist yet would have no target),
+    // which is exactly the `forNewSession` distinction the UI renders.
+    this.deps.broadcast?.('remote:hostWaking', {
+      ...(opts.sessionId ? { sessionId: opts.sessionId } : { forNewSession: true }),
+      hostId: remote.hostId,
+      label: remote.label,
+    });
+    this.deps.log?.(`[RemoteWake] waking ${remote.label} (${remote.host}) via ${target.kind} ${forWhat}`);
+
+    const woke = await this.deps.wake(target);
+    if (!woke) {
+      this.deps.log?.(
+        `[RemoteWake] wake failed for ${remote.label}: ${target.kind === 'command' ? target.command : 'magic packet'}`
+      );
+    }
+
+    const ready = await this.deps.waitUntilReady(remote, { timeoutMs: opts.timeoutMs });
+    if (!ready) {
+      this.deps.log?.(
+        `[RemoteWake] ${remote.label} did not come back — ${opts.forNewSession ? 'the session was not started' : 'input stays buffered'}`
+      );
+      this.deps.broadcast?.('remote:hostWakeFailed', {
+        ...(opts.sessionId ? { sessionId: opts.sessionId } : { forNewSession: true }),
+        hostId: remote.hostId,
+        label: remote.label,
+      });
+      state.probedAt = 0;
+      state.reachable = undefined;
+      return false;
+    }
+
+    state.reachable = true;
+    state.probedAt = Date.now();
+    return true;
+  }
+
  private _state(sessionId: string): WakeState {
    let state = this.states.get(sessionId);
    if (!state) {
@@ -675,7 +802,7 @@ export function createDefaultRemoteWakeDeps(overrides: Partial<RemoteWakeDeps> =
  return {
    probe: probeRemoteHostReachable,
    wake: (target) => (target.kind === 'command' ? runRemoteWakeCommand(target.command) : sendWakePackets(target.macs)),
-    waitUntilReady: (remote) => waitUntilRemoteReady(remote),
+    waitUntilReady: (remote, opts) => waitUntilRemoteReady(remote, opts),
    delay,
    ...overrides,
  };
@@ -326,9 +326,16 @@ Object.assign(CodemanApp.prototype, {
   */
  _onRemoteHostWaking(data) {
    const label = data && data.label ? data.label : 'Remote host';
+    // A create-path wake (the user pressed Run / Attach) has no session yet, so
+    // nothing is queued behind it — the wording has to say what actually happens.
+    const forNewSession = Boolean(data && data.forNewSession);
    // Long enough to cover the wake + attach (~10s measured on a warm S3), and it
    // is replaced by `remote:sessionReconnected` the moment the pane is back.
-    this.showToast(`Waking ${label} … input is queued`, 'info', { duration: 12000 });
+    this.showToast(
+      forNewSession ? `Waking ${label} … the session starts when it is back` : `Waking ${label} … input is queued`,
+      'info',
+      { duration: 12000 }
+    );
    const state = this._hostWake;
    if (!state || !data || state.sessionId !== data.sessionId) return;
    state.waking = true;
@@ -340,7 +347,14 @@ Object.assign(CodemanApp.prototype, {
  /** SSE `remote:hostWakeFailed` — the host did not come back in time. */
  _onRemoteHostWakeFailed(data) {
    const label = data && data.label ? data.label : 'Remote host';
-    this.showToast(`${label} did not wake up — queued input is still held`, 'error', { duration: 15000 });
+    const forNewSession = Boolean(data && data.forNewSession);
+    this.showToast(
+      forNewSession
+        ? `${label} did not wake up — no session was started`
+        : `${label} did not wake up — queued input is still held`,
+      'error',
+      { duration: 15000 }
+    );
    const state = this._hostWake;
    if (!state || !data || state.sessionId !== data.sessionId) return;
    state.waking = false;
@@ -28,6 +28,7 @@ import {
  type GrokConfig,
  type DeepSeekConfig,
  type OmpConfig,
+  type RemoteHost,
 } from '../../types.js';
 import { Session, isAltScreenStripMode, isExternalCliMode, isMuxAltScreenOnlyStripMode } from '../../session.js';
 import { SseEvent } from '../sse-events.js';
@@ -67,7 +68,12 @@ import {
  type WaitSignal,
  type SignalWaitResult,
 } from '../session-wait-registry.js';
-import { RemoteWakeRegistry, createDefaultRemoteWakeDeps } from '../../remote-wake.js';
+import {
+  RemoteWakeRegistry,
+  REMOTE_WAKE_REQUEST_READY_TIMEOUT_MS,
+  createDefaultRemoteWakeDeps,
+  type WakeableRemote,
+} from '../../remote-wake.js';
 import { clampWaitMs, MAX_BUFFER_SCAN_BYTES } from '../../config/agent-wait.js';
 import {
  autoConfigureRalph,
@@ -816,6 +822,22 @@ export function resolveOmpConfigForCreate(
  return resolvedId ? { ...ompConfig, resumeSessionId: resolvedId } : ompConfig;
 }

+/**
+ * `RemoteHost` → the wake registry's host shape. They differ in one field name only
+ * (`id` in host config vs `hostId` on a session's `remote`), but the rename is load-
+ * bearing: the registry keys its per-host wake state on `hostId`.
+ */
+function wakeableHost(host: RemoteHost): WakeableRemote {
+  return {
+    hostId: host.id,
+    label: host.label,
+    host: host.host,
+    port: host.port,
+    wakeMac: host.wakeMac,
+    wakeCommand: host.wakeCommand,
+  };
+}
+
 export function registerSessionRoutes(
  app: FastifyInstance,
  ctx: SessionPort & EventPort & ConfigPort & InfraPort & AuthPort & TabLayoutPort,
@@ -928,6 +950,19 @@ export function registerSessionRoutes(
      const { hostId, remoteSessionName } = body.attachRemoteSession;
      const host = (await readRemoteHosts(CODEMAN_CONFIG_DIR)).find((item) => item.id === hostId);
      if (!host) return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Remote host not found');
+      // An explicit wake request is the only thing that may wake a host, and the user
+      // pressing Attach IS one (see quick-start for the same gate, and
+      // `remote-wake.ts` for what must never call this). Without it a sleeping host
+      // answers with an ssh failure that blames anything but the machine being asleep.
+      const hostWake = await remoteWake.ensureHostAwake(wakeableHost(host), {
+        timeoutMs: REMOTE_WAKE_REQUEST_READY_TIMEOUT_MS,
+      });
+      if (hostWake === 'failed') {
+        return createErrorResponse(
+          ApiErrorCode.OPERATION_FAILED,
+          `${host.label} did not come back after a wake-on-LAN request — nothing was attached`
+        );
+      }
      workingDir = `${host.username}@${host.host}:${remoteSessionName}`;
      remote = toAttachedSessionRemote(host, remoteSessionName, workingDir);
    }
@@ -3272,11 +3307,39 @@ export function registerSessionRoutes(
        );
      }

+      // The user pressing "Run" on a case whose host is asleep IS an explicit wake
+      // request (docs/remote-sessions.md §Wake-on-LAN), and the tmux probe below would
+      // otherwise fail with "could not verify tmux on remote host …" — an ssh failure
+      // that blames tmux for a machine that is merely suspended. Wired HERE, in the HTTP
+      // route, and deliberately NOT in the shared session service: `cron-service.ts`
+      // builds sessions through the service, and a wake down there would re-wake the
+      // host on every schedule (the failure invariant #1 exists to prevent).
+      const hostWake = await remoteWake.ensureHostAwake(wakeableHost(host), {
+        timeoutMs: REMOTE_WAKE_REQUEST_READY_TIMEOUT_MS,
+      });
+      if (hostWake === 'failed') {
+        return createErrorResponse(
+          ApiErrorCode.OPERATION_FAILED,
+          `${host.label} did not come back after a wake-on-LAN request — the session was not started`
+        );
+      }
+
      // tmux is a hard prerequisite on the remote host (the agent runs inside a remote
      // tmux server so it survives ssh drops). Probe before spawning so a missing tmux
      // surfaces a clear, structured error instead of a dead "tmux: command not found" pane.
      const tmuxCheck = await checkRemoteTmuxAvailable(host);
      if (!tmuxCheck.ok) {
+        // An unreachable host and a host without tmux fail the same way over ssh, so the
+        // probe's own message would send the user hunting for a tmux install. Ask the
+        // registry (which just probed, when it woke the host) which of the two it is.
+        if (!(await remoteWake.checkHostReachable(wakeableHost(host)))) {
+          return createErrorResponse(
+            ApiErrorCode.OPERATION_FAILED,
+            hostWake === 'no-target'
+              ? `${host.label} (${host.host}) is not reachable, and this host has no wake-on-LAN target — configure a MAC address or a wake command first`
+              : `${host.label} (${host.host}) is not reachable`
+          );
+        }
        return createErrorResponse(ApiErrorCode.OPERATION_FAILED, tmuxCheck.error || 'remote host is missing tmux');
      }