From 43d4be8eeb938448ddd7e982a327537d78993803 Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Wed, 23 Sep 2026 11:36:49 +0200 Subject: [PATCH] fix(session): merge-time fixes for the dead-pane resume pin (#467) - test/setup.ts strips CLAUDE_CONFIG_DIR (pinned in test-env-isolation), so transcript-fixture tests such as session-custom-model-restart no longer go red on a machine that exports it for a separate Claude account (#255). - The vanished-tmux-session branch of _setupOrAttachMuxSession() relaunches the CLI through createSession() just like a failed respawn, so it now takes the same resume pin. A genuinely new session is unaffected. - After a dead-pane respawn of a fallback-chain CLI, _claudeSessionId names the conversation the walk actually pinned instead of the chain tail, which the walk may have passed over for lack of a transcript. - _claudeConfigDir() trims the override like claudeProjectsDir() does. - The remote-reattach test is labelled as documentation, since the pin builder's own remote guard would make it pass either way. - CLAUDE.md: the create-path pin persists through toState() as resumeSessionId, and the end of the walk adds no pin rather than clearing the launch seed. Co-Authored-By: Claude Opus 5.5 (1M context) --- CLAUDE.md | 2 +- src/session.ts | 51 ++++++++++---- test/respawn-session-id-collision.test.ts | 84 ++++++++++++++++++++++- test/setup.ts | 6 ++ test/test-env-isolation.test.ts | 1 + 5 files changed, 130 insertions(+), 14 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 11d65e47..30a4fcbd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -231,7 +231,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **DeepSeek web UI** (`POST`/`GET`/`DELETE /api/deepseek/web`, `deepseek-web-server.ts`): the Run menu's "DeepSeek web UI..." entry supervises ONE background `dsh web` child process, deliberately **NOT a shell session**. The session version worked and was still wrong in use: it put a terminal tab on screen next to the web tab the user actually asked for, every single time, and nothing about a long-lived HTTP server needs to be a tab. ⚠️ What a session gave for free now has to be paid for explicitly, and every piece is load-bearing: **exactly one** server (a second click REUSES it rather than racing it for a port, which two sessions structurally could not do), **restarted when the browser authority changes** (`--trusted-host` fences dsh's `/api` against the browser authority, and a Codeman reachable at both loopback and a tailnet name has two, so whoever asks last wins: the asker is by definition the origin about to load the page), **killed on shutdown** (`stopDeepSeekWeb()` in the server teardown, because the child is detached so its whole plugin tree can be signalled at once, which also means it would OUTLIVE Codeman and hold its port against the next start), and **failures returned to the caller**, since with no tab there is nowhere for a stack trace to land. ⚠️ The port search starts at dsh's own default 3080 and walks 40, never fixed: that default is precisely the port most likely to be taken already by the user's own `dsh web`, and hardcoding it killed this feature with EADDRINUSE once. Free-port detection BINDS rather than connects (a connect probe cannot tell "free" from "listening but not answering yet"), so it is racy by nature and the caller still waits for the server to really answer before reporting success. ⚠️ Both `POST` and `DELETE` sit at the **same privilege bar as the profile installer** (`canUsernameRunPrivilegedCommands`) even though the action reads as "open a page": booting a dsh profile executes the plugin code in it, and the server is a single shared instance, so stopping it in multi-user mode takes it out from under other users' tabs. -**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`, design doc `docs/custom-model-endpoints-plan.md`; full stack — settings-panel CRUD + the Run-menu picker, on top of the backend below): points a session at a user-configured custom OpenAI-compatible endpoint — local (llama.cpp, DGX Spark, Strix Halo) or cloud (Azure AI Foundry, OpenRouter) — instead of its harness's native cloud backend. Endpoints are a read/write-array store (`custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`) discovered via `GET /v1/models`; `CustomModelHost.authStyle` is `'bearer'` (default, `Authorization: Bearer`) or `'api-key'` (Azure's convention) — **never both**, live-tested against a real server: sending both headers on one request reliably hangs it indefinitely, reproduced 3×. ⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified. ⚠️ **Two launch paths, chosen by mechanism, not preference — see the second paragraph below for why**: opencode/codex/gemini/pi/grok/deepseek/omp apply the selection ONE-SHOT, before the session/process ever exists, with no restart at all; claude alone still applies a selection by **restarting the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` relaunches a CLI in an existing pane, so a CLI whose launch declares a `fallback` chain (claude) gets a conversation pinned as `resumeSessionId` for that one respawn: `--session-id ` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume || --session-id ` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. ⚠️ **The dead-pane respawn needs the same pin and shares it** (`_buildRespawnPaneOptionsWithResumePin()` in `session.ts`, used by `restartCli()`, the dead-pane respawn in `_setupOrAttachMuxSession()` and the create-path fallback after a failed respawn): a pane whose agent EXITED owns a transcript too, so recovering one with the bare launch line hit the same refusal and the conversation was stranded behind a tab that looked merely idle. The pin walks three candidates in order — the conversation chain's tail, the launch seed, then the session's own id — and takes the first one a transcript backs, never `_claudeSessionId` (which also holds history-correlated GUESSES keyed on the working directory, and launching from one would open and write to a conversation that was never this pane's). ⚠️ A candidate no transcript backs is passed over, and falling off the end of the walk pins NOTHING: a divergent pin leaves `--session-id ` in the fallback branch, where a failed resume collides all over again, while pinning an id with no transcript prints claude's "No conversation found" into a brand-new session's scrollback and costs the running branch its `nice` priority (`wrapWithNice()` prefixes only the first branch of an `a || b`). ⚠️ Remote and docker sessions are never pinned: their pane commands are already self-healing, the conversation lives on the far side, and a local id resolves to nothing there. ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/` for pi/omp, grok's `[model.codeman-custom]` block name): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600. ⚠️ **Security**: every env var this feature can redirect (`ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`, `CODEX_HOME`, `GROK_HOME`, `HOME` for pi/omp, `OPENCODE_CONFIG_CONTENT`, etc.) is in that CLI's `privilegedEnvKeys` — several of these were reachable via the generic `envOverrides` field's prefix allowlist BEFORE this feature existed (the env allowlist is global and prefix-based, not per-CLI-scoped), so building this surfaced and closed a pre-existing gap rather than opening a new one. `ANTHROPIC_*` is deliberately NOT in claude's `allowedPrefixes` at all — Anthropic-traffic redirection can only happen through this feature's own admin-configured, SSRF-guarded route, never a plain client-supplied `envOverrides`. **Confidence, verified end-to-end against a real llama-swap server via the DYNAMIC `scripts/test-local-llm-harnesses.ts`** (reads the live CLI registry, so a registry change needs zero script edits): claude/opencode/pi/grok/omp **PASS**; codex config structure is correct, and codex only speaks the Responses API since Feb 2026 (`wire_api = "responses"`) — re-verified live against a llama-swap deployment that DOES answer `/v1/responses` (an earlier test's harder failure against a different deployment does not reproduce everywhere): a plain, no-tool-call chat turn gets a real reply, but a real tool-call attempt came back as `agent_message` TEXT (the tool-call JSON printed as the answer) rather than an executable `function_call` item — confirmed via `codex exec --json`'s raw event stream. Tool execution is what makes codex a coding agent, so it remains not usable for real work either way, just with a more precise failure mode than a flat protocol break; gemini fails with `Invalid auth method selected` (an undocumented `GATEWAY` AuthType gemini-cli selects once `GOOGLE_GEMINI_BASE_URL` is set — unresolved after real investigation); deepseek's originally-reported `HTTP_404` is root-caused and fixed — its bundled `@deepseek-ai/dsh-llm-deepseek` module builds `${DEEPSEEK_BASE_URL}/chat/completions` with no `/v1` of its own (confirmed by installing the real package and reading its source), so a new `appendV1Suffix` flag on its registry entry (alone — claude/gemini must not get it) runs `endpoint.baseUrl` through `withV1Suffix()` before writing it, live-confirmed against llama-swap (`.../chat/completions` 404s, `.../v1/chat/completions` succeeds) though not yet re-run through an actual `dsh` binary, which isn't installable in this environment; antigravity has no known mechanism at all. See the confidence table in `docs/custom-model-endpoints-plan.md` for the full detail on each. ⚠️ **The Run-menu picker generates entries from `window.__codemanCustomModelClis`** (`server.ts`, injected at page render from `enabledClis().filter(kind==='agent' && customModelInjection.kind!=='unsupported')`, JSON-escaped against a literal `` via the exported `escapeScriptJson()` since `label` is a user-`clis.json`-settable string unlike the neighbouring booleans-only `__codemanCliAvailable`), never a hardcoded per-CLI id list in the frontend — the same "no branching on CLI id outside stock.ts" discipline the registry itself enforces. One entry per (capable, INSTALLED CLI, saved endpoint) pair, e.g. "Claude Code (llama.cpp)", filtered through `isCliAvailable()` like the stock entries. Clicking one calls `selectCustomModelEntry(mode, endpointId)` (`session-ui.js`), which re-fetches the endpoint (never trusts anything cached from the dropdown's render — the 5-minute sweep below or a settings edit may have changed it since) and decides the model: exactly one discovered model launches straight away, two or more open `#customModelPickModal` to ask, with `defaultModelId` marked but never auto-chosen (asking exists so ONE launch can deliberately differ from the saved default). ⚠️ **The picker promotes exactly one row to the top**: whichever model llama-swap reports `ready` right now (tagged "Currently loaded", queried via `GET /api/model-endpoints/:id/running-status`, client-side bounded to ~800ms via `Promise.race` so a sleeping/firewalled endpoint cannot leave the modal invisible for the route's own 5s server-side timeout) beats a merely remembered choice, and — only when nothing is currently loaded — the model actually launched last for this exact (harness, endpoint) pair (tagged "Last used", read from the per-device `codeman:customModelLastUsed::` localStorage key). Neither tag reorders past the top, and the "Default" pill is a SEPARATE span rather than a third value of the same slot, so a promoted row that is also the endpoint's `defaultModelId` shows both (on a single-purpose GPU box that is the common case; an exclusive slot silently dropped the Default marking for exactly that row). "Last used" is written by `_runCustomModelEntryViaRestart` (claude) and `_quickStartWithCustomModelConfirm` (every one-shot launch; `runCustomModelEntry` itself only dispatches between the two) only once the model is actually applied — never on the mere click — because a context-window-warning decline means this exact model cannot work with this CLI at all, and promoting a model that cannot launch would be actively wrong, not just premature. Either way the actual launch (`runCustomModelEntry`) routes through `run()` itself via a temporary `_runMode` swap — never `setRunMode()`, which would persist it as the user's new default — rather than a parallel dispatch table, which is what gives a custom-model launch the same `_runInFlight` lock every other Run click gets and means a CLI whose injection recipe lands later needs no update here. It then GETs `/api/sessions/:id/wait?until=idle&timeout=20000` on that session BEFORE applying — measured live, a freshly launched CLI reports itself `busy` for its own startup (boot spinner, workspace-trust check) well before the apply call would otherwise reach it, and the apply route's `isBusy()` guard correctly can't tell that apart from a real turn in progress, so every fresh launch failed with `SESSION_BUSY` until this wait was added. A timeout there is a normal 200 per the wait endpoint's own contract, never an error, so a session still busy after 20s just reaches the apply call anyway and gets that route's own honest error. It then calls `POST /api/sessions/:id/custom-model` on the session `run()` produced, guarded by snapshotting `activeSessionId` before the call and requiring it to have actually changed after — every `run*()` handles its own failure internally and returns normally rather than throwing, so a declined/failed launch must not silently re-point and restart whatever session was already open. ⚠️ The apply call reads the response body itself (`_api()`) rather than `_apiJson()`, which unwraps success but silently discards a failure body — losing the one thing (`error`) that distinguishes "still busy", "remote/Docker session" and everything else the route can report (⚠️ neither route validates `modelId` against the endpoint's discovered list, deliberately: discovery can be up to 5 minutes stale, so a 400 there would refuse a launch that works — a typo'd id fails on the CLI's own first request instead); the resulting toast is `type: 'error'` with an explicit `duration: 0` (no auto-dismiss, an explicit close button) at that one call site — not a blanket sticky-error default, which stacked unbounded on `.toast-container` with no cap or eviction — precisely so a message worth diagnosing survives long enough to be read instead of vanishing on the usual 3s timer. Entries are hidden for a remote/docker active case (the apply route refuses both) and for an endpoint with no discovered models at all (nothing to launch with). ⚠️ **Every saved endpoint's models also re-discover themselves automatically**, a `this.cleanup.setInterval` in `server.ts` (`CUSTOM_MODEL_REDISCOVER_INTERVAL_MS`, 5 minutes, off under `testMode` like the Codex plan-usage poll beside it) calling the exported `refreshAllCustomModelHosts()` (`custom-model-routes.ts`) — one endpoint unreachable on a cycle never blocks the others, and a read-modify-write PER HOST (re-reading the store before each splice, keyed by id) means an admin's concurrent edit or delete wins over a sweep that started before it, never the reverse. +**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`, design doc `docs/custom-model-endpoints-plan.md`; full stack — settings-panel CRUD + the Run-menu picker, on top of the backend below): points a session at a user-configured custom OpenAI-compatible endpoint — local (llama.cpp, DGX Spark, Strix Halo) or cloud (Azure AI Foundry, OpenRouter) — instead of its harness's native cloud backend. Endpoints are a read/write-array store (`custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`) discovered via `GET /v1/models`; `CustomModelHost.authStyle` is `'bearer'` (default, `Authorization: Bearer`) or `'api-key'` (Azure's convention) — **never both**, live-tested against a real server: sending both headers on one request reliably hangs it indefinitely, reproduced 3×. ⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified. ⚠️ **Two launch paths, chosen by mechanism, not preference — see the second paragraph below for why**: opencode/codex/gemini/pi/grok/deepseek/omp apply the selection ONE-SHOT, before the session/process ever exists, with no restart at all; claude alone still applies a selection by **restarting the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` relaunches a CLI in an existing pane, so a CLI whose launch declares a `fallback` chain (claude) gets a conversation pinned as `resumeSessionId` for that one respawn: `--session-id ` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume || --session-id ` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. ⚠️ **The dead-pane respawn needs the same pin and shares it** (`_buildRespawnPaneOptionsWithResumePin()` in `session.ts`, used by `restartCli()`, the dead-pane respawn in `_setupOrAttachMuxSession()`, and its create path when that path RELAUNCHES a CLI: after a failed respawn, or when tmux lost the whole session rather than the pane): a pane whose agent EXITED owns a transcript too, so recovering one with the bare launch line hit the same refusal and the conversation was stranded behind a tab that looked merely idle. The pin walks three candidates in order — the conversation chain's tail, the launch seed, then the session's own id — and takes the first one a transcript backs, never `_claudeSessionId` (which also holds history-correlated GUESSES keyed on the working directory, and launching from one would open and write to a conversation that was never this pane's). ⚠️ The create-path pin is written to `_resumeSessionId` as well, so unlike `restartCli()`'s one-respawn pin it PERSISTS through `toState()` as `resumeSessionId`: that field means "what the user asked to resume at creation, or what recovery pinned", and after a dead-pane respawn `_claudeSessionId` names whatever the walk actually pinned rather than the chain tail. ⚠️ A candidate no transcript backs is passed over, and falling off the end of the walk ADDS no pin (the options keep whatever launch seed they already carried): a divergent pin leaves `--session-id ` in the fallback branch, where a failed resume collides all over again, while pinning an id with no transcript prints claude's "No conversation found" into a brand-new session's scrollback and costs the running branch its `nice` priority (`wrapWithNice()` prefixes only the first branch of an `a || b`). ⚠️ Remote and docker sessions are never pinned: their pane commands are already self-healing, the conversation lives on the far side, and a local id resolves to nothing there. ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/` for pi/omp, grok's `[model.codeman-custom]` block name): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600. ⚠️ **Security**: every env var this feature can redirect (`ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`, `CODEX_HOME`, `GROK_HOME`, `HOME` for pi/omp, `OPENCODE_CONFIG_CONTENT`, etc.) is in that CLI's `privilegedEnvKeys` — several of these were reachable via the generic `envOverrides` field's prefix allowlist BEFORE this feature existed (the env allowlist is global and prefix-based, not per-CLI-scoped), so building this surfaced and closed a pre-existing gap rather than opening a new one. `ANTHROPIC_*` is deliberately NOT in claude's `allowedPrefixes` at all — Anthropic-traffic redirection can only happen through this feature's own admin-configured, SSRF-guarded route, never a plain client-supplied `envOverrides`. **Confidence, verified end-to-end against a real llama-swap server via the DYNAMIC `scripts/test-local-llm-harnesses.ts`** (reads the live CLI registry, so a registry change needs zero script edits): claude/opencode/pi/grok/omp **PASS**; codex config structure is correct, and codex only speaks the Responses API since Feb 2026 (`wire_api = "responses"`) — re-verified live against a llama-swap deployment that DOES answer `/v1/responses` (an earlier test's harder failure against a different deployment does not reproduce everywhere): a plain, no-tool-call chat turn gets a real reply, but a real tool-call attempt came back as `agent_message` TEXT (the tool-call JSON printed as the answer) rather than an executable `function_call` item — confirmed via `codex exec --json`'s raw event stream. Tool execution is what makes codex a coding agent, so it remains not usable for real work either way, just with a more precise failure mode than a flat protocol break; gemini fails with `Invalid auth method selected` (an undocumented `GATEWAY` AuthType gemini-cli selects once `GOOGLE_GEMINI_BASE_URL` is set — unresolved after real investigation); deepseek's originally-reported `HTTP_404` is root-caused and fixed — its bundled `@deepseek-ai/dsh-llm-deepseek` module builds `${DEEPSEEK_BASE_URL}/chat/completions` with no `/v1` of its own (confirmed by installing the real package and reading its source), so a new `appendV1Suffix` flag on its registry entry (alone — claude/gemini must not get it) runs `endpoint.baseUrl` through `withV1Suffix()` before writing it, live-confirmed against llama-swap (`.../chat/completions` 404s, `.../v1/chat/completions` succeeds) though not yet re-run through an actual `dsh` binary, which isn't installable in this environment; antigravity has no known mechanism at all. See the confidence table in `docs/custom-model-endpoints-plan.md` for the full detail on each. ⚠️ **The Run-menu picker generates entries from `window.__codemanCustomModelClis`** (`server.ts`, injected at page render from `enabledClis().filter(kind==='agent' && customModelInjection.kind!=='unsupported')`, JSON-escaped against a literal `` via the exported `escapeScriptJson()` since `label` is a user-`clis.json`-settable string unlike the neighbouring booleans-only `__codemanCliAvailable`), never a hardcoded per-CLI id list in the frontend — the same "no branching on CLI id outside stock.ts" discipline the registry itself enforces. One entry per (capable, INSTALLED CLI, saved endpoint) pair, e.g. "Claude Code (llama.cpp)", filtered through `isCliAvailable()` like the stock entries. Clicking one calls `selectCustomModelEntry(mode, endpointId)` (`session-ui.js`), which re-fetches the endpoint (never trusts anything cached from the dropdown's render — the 5-minute sweep below or a settings edit may have changed it since) and decides the model: exactly one discovered model launches straight away, two or more open `#customModelPickModal` to ask, with `defaultModelId` marked but never auto-chosen (asking exists so ONE launch can deliberately differ from the saved default). ⚠️ **The picker promotes exactly one row to the top**: whichever model llama-swap reports `ready` right now (tagged "Currently loaded", queried via `GET /api/model-endpoints/:id/running-status`, client-side bounded to ~800ms via `Promise.race` so a sleeping/firewalled endpoint cannot leave the modal invisible for the route's own 5s server-side timeout) beats a merely remembered choice, and — only when nothing is currently loaded — the model actually launched last for this exact (harness, endpoint) pair (tagged "Last used", read from the per-device `codeman:customModelLastUsed::` localStorage key). Neither tag reorders past the top, and the "Default" pill is a SEPARATE span rather than a third value of the same slot, so a promoted row that is also the endpoint's `defaultModelId` shows both (on a single-purpose GPU box that is the common case; an exclusive slot silently dropped the Default marking for exactly that row). "Last used" is written by `_runCustomModelEntryViaRestart` (claude) and `_quickStartWithCustomModelConfirm` (every one-shot launch; `runCustomModelEntry` itself only dispatches between the two) only once the model is actually applied — never on the mere click — because a context-window-warning decline means this exact model cannot work with this CLI at all, and promoting a model that cannot launch would be actively wrong, not just premature. Either way the actual launch (`runCustomModelEntry`) routes through `run()` itself via a temporary `_runMode` swap — never `setRunMode()`, which would persist it as the user's new default — rather than a parallel dispatch table, which is what gives a custom-model launch the same `_runInFlight` lock every other Run click gets and means a CLI whose injection recipe lands later needs no update here. It then GETs `/api/sessions/:id/wait?until=idle&timeout=20000` on that session BEFORE applying — measured live, a freshly launched CLI reports itself `busy` for its own startup (boot spinner, workspace-trust check) well before the apply call would otherwise reach it, and the apply route's `isBusy()` guard correctly can't tell that apart from a real turn in progress, so every fresh launch failed with `SESSION_BUSY` until this wait was added. A timeout there is a normal 200 per the wait endpoint's own contract, never an error, so a session still busy after 20s just reaches the apply call anyway and gets that route's own honest error. It then calls `POST /api/sessions/:id/custom-model` on the session `run()` produced, guarded by snapshotting `activeSessionId` before the call and requiring it to have actually changed after — every `run*()` handles its own failure internally and returns normally rather than throwing, so a declined/failed launch must not silently re-point and restart whatever session was already open. ⚠️ The apply call reads the response body itself (`_api()`) rather than `_apiJson()`, which unwraps success but silently discards a failure body — losing the one thing (`error`) that distinguishes "still busy", "remote/Docker session" and everything else the route can report (⚠️ neither route validates `modelId` against the endpoint's discovered list, deliberately: discovery can be up to 5 minutes stale, so a 400 there would refuse a launch that works — a typo'd id fails on the CLI's own first request instead); the resulting toast is `type: 'error'` with an explicit `duration: 0` (no auto-dismiss, an explicit close button) at that one call site — not a blanket sticky-error default, which stacked unbounded on `.toast-container` with no cap or eviction — precisely so a message worth diagnosing survives long enough to be read instead of vanishing on the usual 3s timer. Entries are hidden for a remote/docker active case (the apply route refuses both) and for an endpoint with no discovered models at all (nothing to launch with). ⚠️ **Every saved endpoint's models also re-discover themselves automatically**, a `this.cleanup.setInterval` in `server.ts` (`CUSTOM_MODEL_REDISCOVER_INTERVAL_MS`, 5 minutes, off under `testMode` like the Codex plan-usage poll beside it) calling the exported `refreshAllCustomModelHosts()` (`custom-model-routes.ts`) — one endpoint unreachable on a cycle never blocks the others, and a read-modify-write PER HOST (re-reading the store before each splice, keyed by id) means an admin's concurrent edit or delete wins over a sweep that started before it, never the reverse. **Everything below landed after the initial backend + picker cut, each confirmed live against a real llama-swap deployment.** ⚠️ **llama-swap runs one model at a time, and switching can disrupt ANOTHER live session** — before applying, both apply routes call llama-swap's own `GET /running` (feature-detected via `getLlamaSwapStatus()`, `custom-model-routes.ts`; a plain llama.cpp/OpenAI-compatible server has no such endpoint and is simply never checked). If a different model is loaded and ready AND another live session's own selection is using it, the apply returns `{requiresConfirmation, currentlyLoadedModel, affectedSessions}` instead of switching silently; retrying with `confirmedSwap: true` skips the check, and switching with nothing else affected proceeds immediately. ⚠️ **The swap question and the context-floor question below have SEPARATE flags** (`confirmedSwap`, `confirmedContext`), because the context check runs first and while both shared one `confirmed` a user who clicked past a too-small context silently consented to evicting another session's model too; the legacy `confirmed` still means both, since it shipped in the HTTP-API-only cut. llama-swap also has no dedicated "switch model" endpoint — the only thing that actually starts a swap is a real inference request naming the model (confirmed live: applying a selection alone never reached llama-swap's own logs, since nothing had asked it to load anything) — so both routes also fire `triggerLlamaSwapLoad()`, a fire-and-forget `POST /v1/chat/completions` with `max_tokens: 1`, whenever the target model isn't already loaded and ready. ⚠️ **That launch-time check cannot catch a swap caused by a DIFFERENT session's LATER, ordinary use** — confirmed live: a second Codex session picking a different model launched with no warning at all (nothing conflicted at that exact instant), yet it silently evicted the first session's model regardless, since llama-swap has no push notification of its own. `detectCustomModelSwapDisplacements()` (`custom-model-routes.ts`) is a separate periodic sweep (`server.ts`, `CUSTOM_MODEL_SWAP_CHECK_INTERVAL_MS` = 20s) that compares each live custom-model session's own `modelId` against what `/running` actually reports loaded, broadcasting a `custom-model:swapped-out` SSE event — shown as a global toast, never tied to the displaced session's own tab, since the whole point is telling the user before they type into it — the first time a mismatch appears, via a caller-owned de-dupe `Set` cleared once that session's own model is loaded and ready again so a later, genuinely new displacement notifies again rather than staying silently un-notified forever after the first one. ⚠️ **Context length is read from the REAL launch command, never `/props`** — `/props?model=`'s `default_generation_settings.n_ctx` was confirmed live to report a `--fit-ctx`-launched backend's theoretical/trained maximum rather than the real runtime-configured size (a measured 154112-vs-16384 discrepancy, caught only because the unfixed value still overflowed), so discovery parses the actual configured size straight out of `/running`'s own `cmd` field instead (`parseCtxFromCmd`: `--fit-ctx ` first, then plain llama.cpp `-c`/`--ctx-size`), falling back to `/props` only when `cmd` states no recognizable flag at all. ⚠️ **Claude alone gets a context-window FLOOR check, on top of the ceiling `contextLengthVar` already fixes** — `exceedsSafeContextFloor()` (gated on the registry declaring `contextLengthVar`, so a no-op for every other CLI by construction) compares a model's discovered context against `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000): confirmed live, twice, that Claude Code's own system prompt and tool schemas cost roughly 36.4K tokens on the very first message, before any conversation history exists to compact, so a smaller real context fails outright regardless of what `CLAUDE_CODE_MAX_CONTEXT_TOKENS` says (that var only controls when HISTORY gets compacted, and there is none yet on message one). Below the floor, the apply returns `{requiresContextWarning, modelId, contextLength, minSafeContextTokens}` instead of launching, shown as an in-app dialog naming the actual fix: give the model an explicit larger `-c`/`--ctx-size` in llama-swap's config instead of relying on `--fit-ctx` auto-fit, which optimizes for the biggest MODEL that fits rather than the biggest CONTEXT. ⚠️ **A fresh, isolated `CLAUDE_CONFIG_DIR` looks like a brand-new Claude Code profile and replays its ENTIRE first-run sequence on every launch** — the theme picker, the security-notes screen, the per-project "trust this folder?" dialog, and (running with a bypass-permissions flag) a one-time warning about it, confirmed live, none of which a real, already-onboarded profile shows again. `skipFirstRunPrompts` (claude's entry only, requires `apiKeyTrustFile` since it reuses the same file) pre-seeds that same "already been through this" state: `hasCompletedOnboarding` and this session's own `projects[workingDir].hasTrustDialogAccepted` merge into the same `.claude.json` the API-key trust file already writes to, and `skipDangerousModePermissionPrompt` merges into `settings.json` (a different file, same corrupt-tolerant merge). ⚠️ **The loading banner shows the REAL backend log line, not a guess, and has no countdown or auto-timeout at all.** `getLatestLlamaSwapLogLine()` holds one `GET /api/events` SSE connection open per endpoint (confirmed live to stay open indefinitely — read past 220KB over 8s with no `done`; idle-closed after 30s via `pruneIdleLlamaSwapLogTails`, same 20s sweep as the swap-displacement check above), parsing `logData` frames and keeping only `source: "upstream"` (the real `llama-server` process's own stdout) lines, never `source: "proxy"` (llama-swap's own request-access log). ⚠️ `GET /logs` — the endpoint this feature's own first cut targeted, since the name suggested it — was confirmed live to carry ONLY the proxy log and never a single backend line, even seconds after a real, verified model swap; caught and corrected by a live check before merge, not after. The banner itself dropped its size-scaled expected-time estimate and matching auto-timeout (a guess dressed up as a fact that could kill a genuinely slow load partway through on slower hardware) for a generic hardware/model-size disclaimer plus a user-driven **Cancel** button (`_showCenterStatus`'s `onCancel` option, a real button distinct from the plain "×" close glyph an `'error'`-type banner gets) that ends the wait and closes the session on the user's own call rather than a guessed deadline. diff --git a/src/session.ts b/src/session.ts index 6a36f253..ea8b22f7 100644 --- a/src/session.ts +++ b/src/session.ts @@ -1875,29 +1875,40 @@ export class Session extends EventEmitter { respawnPaneOptions: import('./mux-interface.js').RespawnPaneOptions; createSessionOptions: import('./mux-interface.js').CreateSessionOptions; spawnErrLabel: string; - }): Promise<{ isRestored: boolean }> { + }): Promise<{ isRestored: boolean; respawnedResumeId?: string; respawnedDeadPane: boolean }> { const mux = this._mux!; - // Verify stale mux session — tmux may have been destroyed (e.g., killed externally) + // Verify stale mux session — tmux may have been destroyed (e.g., killed externally). + // A session that HAD a mux session relaunches its CLI below just like a failed + // respawn does (tmux kill-server, a tmux crash, an external kill-session), so + // its transcript collides with the bare `--session-id` the same way. A + // genuinely new session starts with `_muxSession` null and never sets this. + let muxSessionVanished = false; if (this._muxSession && !mux.muxSessionExists(this._muxSession.muxName)) { console.log('[Session] Stale mux session detected (tmux gone):', this._muxSession.muxName); this._muxSession = null; + muxSessionVanished = true; } // Check if session exists but pane is dead (remain-on-exit keeps it alive) // Respawn the pane instead of creating a whole new session — preserves tmux scrollback let needsNewSession = false; + let respawnedDeadPane = false; + let respawnedResumeId: string | undefined; if (this._muxSession && mux.isPaneDead(this._muxSession.muxName)) { console.log('[Session] Dead pane detected, respawning:', this._muxSession.muxName); // Confirmed dead — safe to resolve/pin now (see `_pinOmpRespawnId()`). // `options.respawnPaneOptions` was built eagerly before this dead-pane // check ran, so it still carries the pre-pin ompConfig; rebuild it. this._pinOmpRespawnId(); - const newPid = await mux.respawnPane(await this._buildRespawnPaneOptionsWithResumePin()); + const respawnOptions = await this._buildRespawnPaneOptionsWithResumePin(); + const newPid = await mux.respawnPane(respawnOptions); if (!newPid) { console.error('[Session] Failed to respawn pane, will create new session'); needsNewSession = true; } else { + respawnedDeadPane = true; + respawnedResumeId = respawnOptions.resumeSessionId; this._pendingEnvUnsets.clear(); // Wait a moment for the respawned process to fully start await new Promise((resolve) => setTimeout(resolve, MUX_STARTUP_DELAY_MS)); @@ -1931,8 +1942,9 @@ export class Session extends EventEmitter { // `this.id` while the CLI resumes the chain tail. The response viewer, // Read My Mind and the unified-list alias map all read `_claudeSessionId` // until the next first-hand hook, so the two have to name the same - // conversation. - if (needsNewSession) { + // conversation. The vanished-tmux-session branch above relaunches for the + // same reason and takes the same pin. + if (needsNewSession || muxSessionVanished) { const pinned = (await this._buildRespawnPaneOptionsWithResumePin()).resumeSessionId; if (pinned) { options.createSessionOptions.resumeSessionId = pinned; @@ -1980,7 +1992,7 @@ export class Session extends EventEmitter { throw spawnErr; } - return { isRestored }; + return { isRestored, respawnedResumeId, respawnedDeadPane }; } /** @@ -2165,9 +2177,10 @@ export class Session extends EventEmitter { * claude prints "No conversation found" into the scrollback of a session that * is brand new, and `wrapWithNice()` prefixes only the FIRST branch of the * rendered `a || b`, so the branch that actually runs loses its priority for - * the life of the session. Falling off the end of the walk therefore pins - * nothing, which is the right answer: with no transcript anywhere there is - * nothing for the bare `--session-id ` to collide with. + * the life of the session. Falling off the end of the walk therefore adds + * no pin (the options keep any launch seed they already carried), which is + * the right answer: with no transcript anywhere there is nothing for the + * bare `--session-id ` to collide with. * * The create route pre-validates a resume id for the same reason, though it * additionally requires the transcript be substantial — here mere existence @@ -2213,7 +2226,10 @@ export class Session extends EventEmitter { /** The session's Claude config dir when it has been relocated (#255), else undefined. */ private _claudeConfigDir(): string | undefined { - return this._envOverrides?.CLAUDE_CONFIG_DIR; + // Trimmed like `claudeProjectsDir()` trims the process-wide override: the + // envOverrides schema validates keys only, and a whitespace-only value would + // otherwise resolve to a relative path and read "no transcript" for everything. + return this._envOverrides?.CLAUDE_CONFIG_DIR?.trim() || undefined; } /** @@ -2537,7 +2553,7 @@ export class Session extends EventEmitter { // If mux wrapping is enabled, create or attach to a mux session if (this._useMux && this._mux) { try { - const { isRestored } = await this._setupOrAttachMuxSession({ + const { isRestored, respawnedResumeId, respawnedDeadPane } = await this._setupOrAttachMuxSession({ // Single source of truth shared with reattachRemote() (COD-108). respawnPaneOptions: this._buildRespawnPaneOptions(), createSessionOptions: { @@ -2584,7 +2600,18 @@ export class Session extends EventEmitter { // persisted chain's tail is that conversation, reported first-hand by // the CLI's own hook, so it outranks every fallback here. A NEW pane has // an empty chain and falls through to the resume/alias fallbacks. - restoredConversation = isRestored ? this._claudeSessionChain[this._claudeSessionChain.length - 1] : undefined; + // + // A dead-pane respawn is NOT that case for a CLI whose relaunch the resume + // pin walk governs (`launch.chain === 'fallback'`): the CLI did stop, and + // the walk may have passed over a chain tail with no transcript behind it, + // so the conversation is whatever the respawn actually resumed. Undefined + // there means the pane launched unpinned, which the fallbacks below name. + const pinGovernsRespawn = respawnedDeadPane && getCli(this.mode)?.launch.chain === 'fallback'; + restoredConversation = pinGovernsRespawn + ? respawnedResumeId + : isRestored + ? this._claudeSessionChain[this._claudeSessionChain.length - 1] + : undefined; this._claudeSessionId = restoredConversation || this._resumeSessionId || diff --git a/test/respawn-session-id-collision.test.ts b/test/respawn-session-id-collision.test.ts index 90591e22..02545f53 100644 --- a/test/respawn-session-id-collision.test.ts +++ b/test/respawn-session-id-collision.test.ts @@ -82,6 +82,32 @@ function failingRespawnMux() { return { mux: mux as unknown as TerminalMultiplexer, calls }; } +/** + * A mux whose tmux lost the WHOLE session (tmux kill-server, a crash, an + * external kill-session), not just the pane. `_setupOrAttachMuxSession()` drops + * its stale handle and goes straight to `createSession()`, relaunching the CLI + * exactly as the failed-respawn fallback does. + */ +function vanishedSessionMux() { + const calls: CreateSessionOptions[] = []; + const respawns: RespawnPaneOptions[] = []; + const mux = { + isAvailable: () => true, + muxSessionExists: () => false, + isPaneDead: () => false, + setAttached: () => {}, + respawnPane: async (options: RespawnPaneOptions) => { + respawns.push(options); + return 4242; + }, + createSession: async (options: CreateSessionOptions) => { + calls.push(options); + return muxSession('codeman-recreated'); + }, + }; + return { mux: mux as unknown as TerminalMultiplexer, calls, respawns }; +} + const CONVERSATION = 'aaaabbbb-cccc-dddd-eeee-ffff00001111'; let configDir: string; @@ -258,6 +284,58 @@ describe('pinning a conversation onto a relaunch', () => { } }); + it('pins the create path when tmux lost the whole session, not just the pane', async () => { + // The stale-session branch nulls the handle and never sets the failed-respawn + // flag, so without its own pin the relaunch carried the bare launch line and + // met the same `--session-id ... already in use` refusal. + giveTranscript(CONVERSATION); + const { mux, calls, respawns } = vanishedSessionMux(); + const session = localSession({ claudeSessionChain: [CONVERSATION] }, mux); + + await session.startInteractive(); + try { + expect(respawns).toHaveLength(0); + expect(calls).toHaveLength(1); + expect(calls[0].resumeSessionId).toBe(CONVERSATION); + expect(session.claudeSessionId).toBe(CONVERSATION); + } finally { + await session.stop(); + } + }); + + it('leaves a genuinely new session unpinned on the create path', async () => { + // No mux handle to begin with, so the stale-session flag is never set and + // the create options keep their original shape. + const { mux, calls } = vanishedSessionMux(); + const session = localSession({ muxSession: undefined }, mux); + giveTranscript(session.id); + + await session.startInteractive(); + try { + expect(calls).toHaveLength(1); + expect(calls[0].resumeSessionId).toBeUndefined(); + } finally { + await session.stop(); + } + }); + + it('names the conversation the dead-pane respawn actually resumed', async () => { + // The chain tail has no transcript, so the walk degrades to the session id. + // The session must then report that id, not the chain tail claude never + // opened: the response viewer, Read My Mind and the alias map all read it. + const { mux, calls } = recordingMux(); + const session = localSession({ claudeSessionChain: [CONVERSATION] }, mux); + giveTranscript(session.id); + + await session.startInteractive(); + try { + expect(calls[0].resumeSessionId).toBe(session.id); + expect(session.claudeSessionId).toBe(session.id); + } finally { + await session.stop(); + } + }); + it('pins nothing for a remote session, whose conversation lives elsewhere', async () => { // The dead-pane respawn is reached by every session shape, unlike // `restartCli()` whose route refuses remote. A local id pinned onto a @@ -311,9 +389,13 @@ describe('pinning a conversation onto a relaunch', () => { expect(calls[0].resumeSessionId).toBeUndefined(); }); - it('pins nothing for a remote reattach, which relaunches no CLI', async () => { + it('documents that a remote reattach carries no pin', async () => { // `reattachRemote()` re-runs the remote session command, which attaches to // the durable remote tmux with the agent still running inside it. + // ⚠️ Documentation, not a regression guard: `reattachRemote()` only runs for + // a remote session, and the pin builder refuses remote sessions on its own, + // so this would still pass if `reattachRemote()` were switched to the pinned + // builder. The builder's remote guard is what the test above pins. giveTranscript(CONVERSATION); const { mux, calls } = recordingMux(); const session = localSession( diff --git a/test/setup.ts b/test/setup.ts index aa2a5945..a7bed796 100644 --- a/test/setup.ts +++ b/test/setup.ts @@ -45,6 +45,12 @@ delete process.env.CODEMAN_GESTURE; // operator who exports it (exactly who the feature is for) would otherwise see the // root-install byte-identity assertions fail. delete process.env.CODEMAN_BASE_URL; +// CLAUDE_CONFIG_DIR (#255) relocates Claude's whole tree, transcripts included, and +// `claudeProjectsDir()` reads it before it ever looks at `homedir()`. A developer who runs +// Codeman against a separate Claude account exports exactly this, and every test that writes +// a transcript fixture under the temp HOME's `~/.claude/projects` then reads "no transcript" +// (found at merge of #467: test/session-custom-model-restart.test.ts went red). +delete process.env.CLAUDE_CONFIG_DIR; // Instance selection is PROCESS-WIDE and is what `src/config/instance.ts` derives // both the data dir and the tmux socket from, so a shell that exports any of these diff --git a/test/test-env-isolation.test.ts b/test/test-env-isolation.test.ts index a0a56d8e..c1097dd8 100644 --- a/test/test-env-isolation.test.ts +++ b/test/test-env-isolation.test.ts @@ -34,6 +34,7 @@ const STRIPPED_ENV_VARS: Array<[name: string, why: string]> = [ ['CODEMAN_INSTANCE', 'moves the data dir to ~/.codeman- and the tmux socket to codeman-'], ['CODEMAN_DATA_DIR', 'ABSOLUTE override: bypasses the temp HOME and points the suite at a real data dir'], ['CODEMAN_TMUX_SOCKET', 'renames the socket resolveTmuxSocketName() returns'], + ['CLAUDE_CONFIG_DIR', 'relocates the Claude tree, so transcript fixtures under the temp HOME read as missing'], ]; const SETUP_SOURCE = readFileSync(fileURLToPath(new URL('./setup.ts', import.meta.url)), 'utf-8');