mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-10-03 14:09:42 +02:00
DevvynandClaude Sonnet 5
7bbe408e44
The loading banner now shows a live countdown against its own timeout (updated every poll, so every second by default) instead of a static "this can take a while" — e.g. "Loading qwen3.8-27b (16.4 GB, typically ~1-3 min) on llama-swap - 47s remaining". If the countdown reaches zero and the model still isn't ready, this is now treated as a real failure rather than a "keep waiting" shrug: - The banner turns into a sticky error (_showCenterStatus gains a `type` option - 'error' drops the spinner and adds a close button, since nothing is "in progress" anymore and a sticky message needs a way to dismiss it), naming the llama-swap server's own logs as where to look for detail. - The session that load was for is closed automatically (closeSession) - requested explicitly: a console left open and pointed at a model that never finished loading is worse than no console at all. Both apply paths now thread the new session's id through to _watchLlamaSwapLoading for this (new required 3rd parameter, after endpointId/modelId). _watchLlamaSwapGeneration's existing stale-call guard extends naturally to this: a superseded call's own eventual timeout recognises it no longer owns the banner and neither shows the error nor closes a session that may by then belong to a different, newer launch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqZeHrRS6DYcGcGX2p9EwG
144 lines
9.3 KiB
Markdown
144 lines
9.3 KiB
Markdown
# Custom Model Endpoints
|
||
|
||
Point a harness at your own OpenAI-compatible server instead of its native cloud backend, for
|
||
one session at a time. "Custom endpoint" covers **local** hardware (llama.cpp, Ollama, vLLM,
|
||
a home GPU rig, DGX Spark, Strix Halo) and **cloud** services (Azure AI Foundry's
|
||
OpenAI-compatible endpoint, OpenRouter, a company gateway) alike, anything answering
|
||
`GET /v1/models` and `POST /v1/chat/completions` in the standard shape.
|
||
|
||
**Off by default.** Turn it on in App Settings → Models → **Custom model endpoints**.
|
||
|
||
## Adding an endpoint
|
||
|
||
Still in App Settings → Models → Custom model endpoints:
|
||
|
||
1. **+ Add endpoint** — give it an id, a label, and the base URL (`http://192.168.1.50:8080`,
|
||
say). An API key is optional; most local servers don't check one.
|
||
2. **Discover** — fetches the endpoint's own model list over `GET /v1/models` and stores it.
|
||
3. Pick a **default model** from what was discovered. This is the model the Run-menu entry
|
||
applies directly when only one model is discovered; with two or more, it's just the one
|
||
pre-marked in the picker dialog described below, not a silent default.
|
||
|
||
Endpoint management is admin-only in multi-user mode, the same as remote hosts and Docker
|
||
hosts — these are machine-level infra, not a per-user setting.
|
||
|
||
**Model lists refresh themselves.** Every saved endpoint is re-discovered automatically every
|
||
5 minutes in the background, so a model the server starts serving later — or stops serving —
|
||
shows up without another manual click of **Discover**. One endpoint being unreachable on a
|
||
given cycle (powered off, wrong network) never blocks the others from refreshing.
|
||
|
||
**Context length is picked up automatically where it can be, safely.** Against a
|
||
llama.cpp/llama-swap server, discovery also learns each *currently loaded* model's real
|
||
context window and applies it to the launched session (Claude Code today — see below), so
|
||
the harness stops assuming a large default window for a model name it doesn't recognise and
|
||
overflowing a much smaller real one. It's deliberately never probed for a model that isn't
|
||
already loaded, since asking a llama-swap server about an unloaded model can trigger an
|
||
actual, slow model swap as a side effect — a model just not currently loaded keeps whatever
|
||
context length an earlier cycle already learned for it instead.
|
||
|
||
## Running a session against one
|
||
|
||
With the setting on and at least one endpoint carrying a discovered model, the **Run**
|
||
dropdown grows a **Custom Endpoints** section: one entry per harness that can redirect to a
|
||
custom endpoint, per saved endpoint, e.g. "Claude Code (llama.cpp)". Picking one starts a
|
||
session on that harness exactly the way its own entry would. It is a one-off "try this
|
||
endpoint" action, not a sticky mode — the plain **Run** button still means "this harness,
|
||
native cloud" afterward, and a fresh session never inherits whatever the last one was
|
||
pointed at.
|
||
|
||
**Which model it uses depends on how many the endpoint has discovered.** With exactly one,
|
||
the session launches straight away on that model — nothing to choose. With two or more, a
|
||
small dialog asks which one to use for this launch before starting the session; the
|
||
endpoint's default model, if set, is marked but not auto-picked, so a launch can deliberately
|
||
use a different one without changing the saved default.
|
||
|
||
**For opencode, Codex, Gemini, Pi, Grok, DeepSeek and OMP, picking an entry launches
|
||
straight onto the endpoint** — no restart, because the endpoint is applied before the
|
||
session's process ever starts. **Claude still restarts the harness's process in place** —
|
||
same tab, same conversation (`--resume`) — after a normal native launch, since that restart
|
||
is far less jarring for Claude than for the other seven, whose own TUI can fully
|
||
reinitialize on a restart. Either way, every supported harness reads its endpoint config at
|
||
process start, never per turn, so there is no live hot-swap while a turn is running.
|
||
|
||
Picking an entry that launches a **brand-new** Claude session waits (up to 20 seconds) for it to
|
||
finish its own startup before applying — a freshly started CLI reports itself as busy for its
|
||
boot sequence, and applying to a genuinely busy session is refused so a real, in-progress
|
||
turn is never interrupted out from under you. A session that is still busy after that wait
|
||
(a very slow-starting CLI, or one you started typing into right away) surfaces that refusal
|
||
as an ordinary error, which now stays on screen with a close button instead of vanishing
|
||
after a few seconds — read it, it names the actual reason rather than a generic failure.
|
||
|
||
Entries are hidden entirely for a session in a **remote (SSH) or Docker case** — support for
|
||
redirecting those hasn't landed yet, see below. The picker also only appears in the desktop
|
||
**Run** dropdown; the phone home screen builds its own run picker separately and does not
|
||
currently offer these entries.
|
||
|
||
**Against llama-swap, applying a selection also starts the actual model load, rather than
|
||
waiting on your first prompt to do it.** llama-swap has no "switch model" button of its own
|
||
— the only thing that starts a swap is a real request naming the model, and confirmed live:
|
||
just applying a selection never reached llama-swap's own logs at all until something asked
|
||
it to load. Picking an entry now also sends the smallest real request that will trigger
|
||
that load, in the background, the moment the target model isn't already loaded and ready.
|
||
|
||
**The centred loading banner shows a live countdown, and a real timeout is an error, not a
|
||
shrug.** When it knows the model's discovered file size (its GB figure, when llama-swap
|
||
states one), it shows both a rough expected-time estimate and a live countdown against it —
|
||
e.g. "Loading qwen3.8-27b (16.4 GB, typically ~1–3 min) on llama-swap — 47s remaining". If
|
||
the countdown reaches zero and the model still isn't ready, the banner turns into a sticky
|
||
error telling you to check the llama-swap server's own logs, and **the session that load was
|
||
for is closed automatically** — a console left open and pointed at a model that never
|
||
finished loading would just be confusing to leave sitting there.
|
||
|
||
**Claude Code specifically gets two extra fixes applied automatically:**
|
||
|
||
- Its discovered context length (see above) is passed through as
|
||
`CLAUDE_CODE_MAX_CONTEXT_TOKENS`, so it doesn't send a full-size prompt against a much
|
||
smaller real local context and overflow it.
|
||
- Its session runs with an isolated `CLAUDE_CONFIG_DIR`, so the injected API key never sits
|
||
in the same directory as a stored claude.ai login — that combination is harmless for actual
|
||
requests (the API key wins) but the CLI still prints a "both claude.ai and
|
||
ANTHROPIC_API_KEY set" warning about it, which this avoids entirely. The isolated directory
|
||
keeps a link back to your real session history so the response viewer and similar features
|
||
still work for that session. That isolated directory starts with no prior approvals of its
|
||
own, so Codeman also pre-approves the injected key the same way answering Claude Code's own
|
||
"Detected a custom API key" prompt once would — without it, that prompt would otherwise
|
||
reappear on every single launch with nobody there to answer it.
|
||
|
||
## Which harnesses actually work
|
||
|
||
| Harness | Status |
|
||
| ------- | ------ |
|
||
| **Claude Code, opencode, Pi, Grok, OMP** | Verified end-to-end against a real local server. |
|
||
| **Codex** | Config is correct, but Codex only speaks the Responses API, which llama.cpp-style servers don't implement. A protocol gap, not a Codeman bug. |
|
||
| **Gemini** | Fails with an auth error gemini-cli raises once redirected. Unresolved; don't rely on it yet. |
|
||
| **DeepSeek** | Reaches the server but gets a consistent 404. Root cause not identified. |
|
||
| **Antigravity** | No known custom-endpoint mechanism at all. Not offered. |
|
||
|
||
Which harnesses show up in the Run-menu picker is read live off Codeman's own CLI registry,
|
||
not a fixed list here, so this table can go stale before this page does — a greyed-out or
|
||
missing entry is the more current answer.
|
||
|
||
## What it does not do
|
||
|
||
- **No remote or Docker sessions yet.** Both restart their agent differently under the hood
|
||
(reattaching a durable tmux session rather than relaunching the process), so redirecting
|
||
them needs its own plumbing that hasn't been built.
|
||
- **No live hot-swap mid-conversation.** Applying a selection always restarts the process.
|
||
- **No button to un-point a session from the UI yet.** Clearing back to native cloud is an
|
||
HTTP call (`POST .../custom-model {"clear": true}`) or deleting the session; the settings
|
||
panel manages saved endpoints, not what a running session is currently pointed at.
|
||
- **Nothing is shared with your real cloud credentials.** The endpoint's own key, if any,
|
||
never touches your Anthropic/OpenAI/Google login — a custom endpoint is a separate,
|
||
explicit choice per session.
|
||
|
||
## Security
|
||
|
||
An endpoint's base URL can't point at a link-local or cloud-metadata address (both at save
|
||
time and against the address it actually resolves to), the same guard Web Tabs uses for
|
||
saved dashboards. Endpoint records and any per-session config files a harness needs are
|
||
written with owner-only permissions. See
|
||
[custom-model-endpoints-plan.md](https://github.com/Ark0N/Codeman/blob/master/docs/custom-model-endpoints-plan.md)
|
||
in the repository for the full design reasoning, including why this feature closed a
|
||
pre-existing gap in how session environment overrides were guarded rather than opening a new
|
||
one.
|