DevvynandClaude Sonnet 5
5c25a52f95
Root cause of every 'Session is busy' apply failure reported from live
testing: a just-launched CLI reports itself 'busy' for its own startup
(boot spinner, workspace-trust check) well before runCustomModelEntry's
apply call could reach it, and the apply route's isBusy() guard correctly
cannot tell that apart from a real turn in progress — it exists precisely
to refuse restarting a session mid-turn, and a fresh boot looks exactly
like one from the outside. Confirmed live: replaying the identical apply
call by hand against the same session, once it had settled, succeeded
immediately.
Fixed by waiting on the session's own readiness signal before applying:
GET /api/sessions/:id/wait?until=idle&timeout=20000, one GET already built
for exactly this ('Agent wait primitives', CLAUDE.md) rather than inventing
a client-side poll loop. A timeout there is a normal 200 per that
endpoint's own contract, never an error, so a session still busy after 20s
just reaches the apply call anyway and gets the route's own honest error —
now visible, since the previous commit made error toasts sticky and
stopped discarding the real error text.
Tests: new case in custom-model-run-menu-ui.test.ts pins the ordering (the
wait call happens, and strictly before the apply call) and its exact query
string.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqZeHrRS6DYcGcGX2p9EwG
6.1 KiB
Custom Model Endpoints
Point a harness at your own OpenAI-compatible server instead of its native cloud backend, for
one session at a time. "Custom endpoint" covers local hardware (llama.cpp, Ollama, vLLM,
a home GPU rig, DGX Spark, Strix Halo) and cloud services (Azure AI Foundry's
OpenAI-compatible endpoint, OpenRouter, a company gateway) alike, anything answering
GET /v1/models and POST /v1/chat/completions in the standard shape.
Off by default. Turn it on in App Settings → Models → Custom model endpoints.
Adding an endpoint
Still in App Settings → Models → Custom model endpoints:
- + Add endpoint — give it an id, a label, and the base URL (
http://192.168.1.50:8080, say). An API key is optional; most local servers don't check one. - Discover — fetches the endpoint's own model list over
GET /v1/modelsand stores it. - Pick a default model from what was discovered. This is the model the Run-menu entry applies directly when only one model is discovered; with two or more, it's just the one pre-marked in the picker dialog described below, not a silent default.
Endpoint management is admin-only in multi-user mode, the same as remote hosts and Docker hosts — these are machine-level infra, not a per-user setting.
Model lists refresh themselves. Every saved endpoint is re-discovered automatically every 5 minutes in the background, so a model the server starts serving later — or stops serving — shows up without another manual click of Discover. One endpoint being unreachable on a given cycle (powered off, wrong network) never blocks the others from refreshing.
Running a session against one
With the setting on and at least one endpoint carrying a discovered model, the Run dropdown grows a Custom Endpoints section: one entry per harness that can redirect to a custom endpoint, per saved endpoint, e.g. "Claude Code (llama.cpp)". Picking one starts a session on that harness exactly the way its own entry would. It is a one-off "try this endpoint" action, not a sticky mode — the plain Run button still means "this harness, native cloud" afterward, and a fresh session never inherits whatever the last one was pointed at.
Which model it uses depends on how many the endpoint has discovered. With exactly one, the session launches straight away on that model — nothing to choose. With two or more, a small dialog asks which one to use for this launch before starting the session; the endpoint's default model, if set, is marked but not auto-picked, so a launch can deliberately use a different one without changing the saved default.
Applying a selection restarts the harness's process in place — same tab, same conversation where the harness supports resuming one, fresh environment. That restart is necessary, not incidental: every supported harness reads its endpoint config at process start, never per turn, so there is no live hot-swap while a turn is running.
Picking an entry that launches a brand-new session waits (up to 20 seconds) for it to finish its own startup before applying — a freshly started CLI reports itself as busy for its boot sequence, and applying to a genuinely busy session is refused so a real, in-progress turn is never interrupted out from under you. A session that is still busy after that wait (a very slow-starting CLI, or one you started typing into right away) surfaces that refusal as an ordinary error, which now stays on screen with a close button instead of vanishing after a few seconds — read it, it names the actual reason rather than a generic failure.
Entries are hidden entirely for a session in a remote (SSH) or Docker case — support for redirecting those hasn't landed yet, see below. The picker also only appears in the desktop Run dropdown; the phone home screen builds its own run picker separately and does not currently offer these entries.
Which harnesses actually work
| Harness | Status |
|---|---|
| Claude Code, opencode, Pi, Grok, OMP | Verified end-to-end against a real local server. |
| Codex | Config is correct, but Codex only speaks the Responses API, which llama.cpp-style servers don't implement. A protocol gap, not a Codeman bug. |
| Gemini | Fails with an auth error gemini-cli raises once redirected. Unresolved; don't rely on it yet. |
| DeepSeek | Reaches the server but gets a consistent 404. Root cause not identified. |
| Antigravity | No known custom-endpoint mechanism at all. Not offered. |
Which harnesses show up in the Run-menu picker is read live off Codeman's own CLI registry, not a fixed list here, so this table can go stale before this page does — a greyed-out or missing entry is the more current answer.
What it does not do
- No remote or Docker sessions yet. Both restart their agent differently under the hood (reattaching a durable tmux session rather than relaunching the process), so redirecting them needs its own plumbing that hasn't been built.
- No live hot-swap mid-conversation. Applying a selection always restarts the process.
- No button to un-point a session from the UI yet. Clearing back to native cloud is an
HTTP call (
POST .../custom-model {"clear": true}) or deleting the session; the settings panel manages saved endpoints, not what a running session is currently pointed at. - Nothing is shared with your real cloud credentials. The endpoint's own key, if any, never touches your Anthropic/OpenAI/Google login — a custom endpoint is a separate, explicit choice per session.
Security
An endpoint's base URL can't point at a link-local or cloud-metadata address (both at save time and against the address it actually resolves to), the same guard Web Tabs uses for saved dashboards. Endpoint records and any per-session config files a harness needs are written with owner-only permissions. See custom-model-endpoints-plan.md in the repository for the full design reasoning, including why this feature closed a pre-existing gap in how session environment overrides were guarded rather than opening a new one.