mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-09-30 12:39:42 +02:00
feat(custom-model): Custom Model Endpoint Profiles (local or cloud, all harnesses)
Point any Codeman-supported harness (Claude, opencode, Codex, Gemini, Pi, Grok, DeepSeek, OMP) at a custom OpenAI-compatible endpoint instead of its native cloud backend, for a given session. Covers local hardware (llama.cpp, Ollama, vLLM, DGX Spark, Strix Halo) and cloud (Azure AI Foundry, OpenRouter). Off by default (customModelEndpointsEnabled, synced, default OFF). - Registry: capabilities.customModelInjection per CLI entry (env / configContentEnv / configDir / unsupported kinds) - Pure injection builder (custom-model-injection.ts) turning an endpoint + model id into the real env vars / config content per CLI - Endpoint store + CRUD routes (custom-model-hosts.ts, custom-model-routes.ts), discovery via GET /v1/models, SSRF-guarded - Session integration: Session.setCustomModel()/restartCli() (POST /api/sessions/:id/custom-model), reusing the existing respawn-pane -k primitive to restart the CLI process with new env - Multi-user hardening: every new redirect-capable env var added to its CLI's privilegedEnvKeys, closing a pre-existing gap where several were already reachable via the generic envOverrides field's prefix allowlist - Standalone scripts/test-local-llm-harnesses.mjs: spawns real CLI binaries against a real endpoint outside the web UI, independent of tmux/sessions - Mock-server contract tests (test/fixtures/mock-openai-server.ts) replaying every CLI's injected values through a real HTTP shape Real end-to-end validation against a live llama-swap server (inside a codeman/agent:llm-test Docker image with all 9 CLI binaries) found and fixed three real bugs before they shipped: - Codex's config.toml schema was wrong ([model].default table instead of a top-level model string + [model_providers.custom]); fixing it then surfaced a genuine, documented protocol incompatibility (Codex only speaks the Responses API since Feb 2026, which llama.cpp/llama-swap don't implement) - Claude Code's async session-title-generation call validates ANTHROPIC_DEFAULT_HAIKU_MODEL against its own internal model list and hangs the whole -p invocation on an unrecognized name; documented for chunk 6, worked around in the standalone script only (--bare is NOT safe for a real interactive session, which needs hooks) - The discovery route's authStyle: 'both' option (send both Authorization and api-key headers) reliably hung a real server; removed the option entirely rather than just changing the default Status: draft. Chunk 6 (frontend toolbar/settings UI) not yet built — see PR.md and deployment_plan.md for the full chunk breakdown and confidence table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017HqNWfmtBU2KN29SvSVWB3
This commit is contained in:
Devvyn
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
a017e9a8e0
commit
41416566aa
@@ -0,0 +1,348 @@
|
||||
# Custom Model Endpoint Profiles (all harnesses, local or cloud)
|
||||
|
||||
## Context
|
||||
|
||||
Devvyn pays for Claude Code but also runs a capable local model behind an
|
||||
OpenAI-compatible server (llama.cpp) — and wants the same mechanism to work
|
||||
against a **cloud** OpenAI-compatible endpoint too (e.g. Azure AI Foundry's
|
||||
OpenAI-compatible inference endpoint, OpenRouter, a self-hosted gateway).
|
||||
Right now every Codeman session mode defaults to its native cloud backend
|
||||
with no way to redirect a session at any other endpoint from the UI — the
|
||||
closest existing precedent is DeepSeek's server-env-sourced
|
||||
`DEEPSEEK_BASE_URL`, which isn't user-facing.
|
||||
|
||||
**Scope note**: this plan originally said "local LLM." It now covers any
|
||||
OpenAI-compatible endpoint the user configures — local (llama.cpp, Ollama,
|
||||
vLLM) or cloud (Azure AI Foundry, OpenRouter, a company gateway). The
|
||||
mechanism is identical (a base URL Codeman probes via `GET /v1/models`); the
|
||||
only real differences are auth-header convention (cloud endpoints often want
|
||||
an `api-key` header, e.g. Azure, rather than `Authorization: Bearer`) and
|
||||
that a cloud "model" may actually be a deployment name distinct from the
|
||||
underlying model family (Azure AI Foundry deployments) — both are called out
|
||||
where they matter below. Naming throughout this plan is **"custom model
|
||||
endpoint,"** not "local model," to keep that scope explicit.
|
||||
|
||||
### Additional use case: on-premises AI hardware
|
||||
|
||||
"Local" isn't limited to a desktop running llama.cpp — a growing category of
|
||||
purpose-built, on-premises AI hardware exists specifically to run a serious
|
||||
model on-site with an OpenAI-compatible server, and this feature is exactly
|
||||
the on-ramp for pointing Codeman at one:
|
||||
|
||||
- **NVIDIA DGX Spark** (and the DGX Spark-class "Spark" mini-supercomputer
|
||||
line) — a compact on-prem inference/training box aimed at running large
|
||||
local models with an OpenAI-compatible API surface.
|
||||
- **AMD "Strix Halo" (Ryzen AI Max)** on-prem AI mini-PCs — unified-memory
|
||||
APU hardware marketed for local LLM inference, typically fronted by
|
||||
llama.cpp/Ollama/vLLM the same way a home server would be.
|
||||
|
||||
Neither needs anything new from this design: both present a standard
|
||||
`/v1/models` + `/v1/chat/completions` OpenAI-compatible surface once the
|
||||
inference server is running, so they're just another `baseUrl` entry in the
|
||||
custom-model-hosts store, same as llama.cpp or a cloud endpoint. The
|
||||
justification for building this generically (rather than hardcoding "point
|
||||
Claude at my llama.cpp box") is precisely this: **the same endpoint registry
|
||||
and per-CLI injection mechanism should work unmodified for any current or
|
||||
future OpenAI-compatible box or service** — a home GPU rig today, a Spark or
|
||||
Strix Halo appliance tomorrow, a company's on-prem inference cluster after
|
||||
that — without Codeman needing to know or care what's actually serving the
|
||||
model on the other end of that URL.
|
||||
|
||||
A concrete example worth naming: **[Ark0N/Qwen5090](https://github.com/Ark0N/Qwen5090)**
|
||||
(from the same GitHub account as this project's owner) is a one-click
|
||||
Windows / one-command Linux installer that stands up Qwen3.8-27B locally on
|
||||
an RTX 5090 (or another RTX 50-series card with ≥24GB) behind an
|
||||
OpenAI-compatible API, served by any of vLLM, NInfer, or llama.cpp — MIT-
|
||||
licensed tooling over Apache-2.0 Qwen weights. It's a direct, ready-made
|
||||
target for this feature: point a custom-model-hosts entry at whichever
|
||||
backend it's running, and it needs nothing further from Codeman's side. It's
|
||||
also notable for already wiring up DeepSeek Harness and Claude Code as
|
||||
coding agents against that local server itself, which is effectively the
|
||||
same "point a Codeman-supported harness at a local endpoint" idea this
|
||||
feature is generalizing — worth using as a real-world reference/test target
|
||||
once chunk 5 (session integration) exists, alongside Devvyn's own llama.cpp
|
||||
box.
|
||||
|
||||
Each harness has its own (different-shaped) mechanism for pointing at a
|
||||
custom OpenAI-compatible base URL + model — env vars for Claude, a JSON
|
||||
config blob for opencode, a TOML file for Codex, etc. Devvyn gave the
|
||||
starting recipes for those three; the rest (Gemini, Pi, Grok, DeepSeek, OMP,
|
||||
Antigravity) were researched for this plan and are flagged by confidence
|
||||
below. A real end-to-end pass against Devvyn's own llama-swap server
|
||||
(`scripts/test-local-llm-harnesses.mjs`, inside a `codeman/agent:llm-test`
|
||||
Docker image with all 9 CLIs installed) then confirmed **claude and
|
||||
opencode work end-to-end**, corrected a real Codex config.toml schema bug
|
||||
the given recipe had (see the Codex row below), and surfaced that Codex's
|
||||
*protocol* — not just its config shape — does not work against a plain
|
||||
OpenAI-Chat-Completions server like llama.cpp/llama-swap at all. Confidence
|
||||
below reflects what was actually observed, not just what was planned.
|
||||
|
||||
The feature must be:
|
||||
|
||||
- **Off by default**, one settings toggle turns it on.
|
||||
- Endpoint entry: user gives a base URL — a LAN address or a cloud URL —
|
||||
plus an optional API key, and Codeman calls `GET <baseUrl>/v1/models` to
|
||||
discover and store the available model (or deployment) list.
|
||||
- A **new toolbar selector** (separate from the existing Run-mode menu, since
|
||||
it's a modifier on top of whichever harness is already selected/running)
|
||||
lets the user pick "Cloud (default)" — the harness's own native backend —
|
||||
or a model discovered from one of the configured custom endpoints.
|
||||
- Picking a custom-endpoint model for an **already-running session restarts
|
||||
that session's CLI process** with the injected env/config pointed at that
|
||||
endpoint (confirmed with Devvyn — these harnesses read endpoint config at
|
||||
process start, not per-turn, so a live hot-swap isn't possible).
|
||||
- **New sessions always default back to the harness's native cloud backend.**
|
||||
A custom-endpoint selection is a per-session override, not a sticky global
|
||||
default — starting a fresh CLI (any mode) always launches against its
|
||||
native backend unless the user explicitly picks a custom endpoint for that
|
||||
new session too. The toolbar selector is scoped to "this session," never
|
||||
carried forward as the default for future sessions.
|
||||
|
||||
This follows the repo's existing data-driven CLI-registry philosophy
|
||||
(`test/cli-registry-no-id-branching.test.ts`): per-CLI behavior is a
|
||||
declared capability, never an `if (mode === 'claude')` branch.
|
||||
|
||||
## Per-CLI injection recipes (confidence-ranked)
|
||||
|
||||
| CLI | Mechanism | Confidence |
|
||||
|---|---|---|
|
||||
| `claude` | Env vars: `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`, `ANTHROPIC_DEFAULT_SONNET_MODEL`/`_HAIKU_MODEL`/`_OPUS_MODEL` (all set to the chosen model/deployment name) | **Verified end-to-end** against a real llama-swap server — a real "hello world" reply came back. ⚠️ Non-interactive (`-p`) invocations also fire an async session-title-generation call that reuses `ANTHROPIC_DEFAULT_HAIKU_MODEL` and validates it against Claude Code's OWN internal recognized-model list, printing `[claude-code:unrecognized_model]` and, in `-p` mode, hanging the whole invocation rather than just warning. `--settings '{"autoTitle":false}'` does NOT stop this (confirmed); `--bare` does (the warning still prints, but the real prompt runs) — but `--bare` ALSO disables hooks, LSP, plugin sync, and CLAUDE.md auto-discovery, so it is only safe for the standalone one-shot test script, NEVER for a real interactive Codeman session (which depends on hooks for idle detection, trust-dialog auto-accept, etc. — see the External CLI modes section of CLAUDE.md). Whether an INTERACTIVE claude session with a custom model hits the same hang (vs. just a background warning) is untested and should be checked before calling chunk 5/6 done for claude |
|
||||
| `opencode` | `OPENCODE_CONFIG_CONTENT` env var (already a registry mechanism, `stock.ts:342`) holding a JSON blob: `{"provider":{"custom":{"options":{"baseURL":...,"apiKey":...},"models":{"<name>":{}}}},"model":"custom/<name>"}` | **Verified by user** |
|
||||
| `codex` | TOML `config.toml`: top-level `model = "<id>"` + `[model_providers.custom]` (`base_url`, `env_key` naming an env var the real API key rides in — never a literal TOML field, since codex's schema has no such field). Written to an isolated dir via `CODEX_HOME` (`stock.ts:405-415`) so the user's own `~/.codex/config.toml` is never touched | **Config STRUCTURE verified** against a real codex binary (an earlier `[model].default` table shape was rejected: "invalid type: map, expected a string" — caught live). **Protocol CONFIRMED BROKEN against llama.cpp/llama-swap**: codex only speaks the Responses API (`wire_api = "responses"`, the only value it accepts since it dropped `"chat"` support in Feb 2026), and a real llama-swap server does not implement `/v1/responses` — a live run against it failed with repeated `Reconnecting...` then `high demand` errors. Codex support therefore needs a Responses-API-compatible endpoint (most local llama.cpp/Ollama/vLLM setups do not qualify); do not present this as working against a generic OpenAI-Chat-Completions box |
|
||||
| `gemini` | Env vars `GOOGLE_GEMINI_BASE_URL` (or `GOOGLE_VERTEX_BASE_URL`) + `GEMINI_API_KEY`; CLI needs a restart to pick them up (matches our restart-on-switch design). Model selection via `--model`/`GEMINI_MODEL`-style override — verify exact var name against the installed `gemini-cli` version before shipping | Web-researched, unverified |
|
||||
| `pi` | Config file `~/.pi/agent/models.json` (hot-reloadable) with a custom provider block: `baseUrl`, `apiKey`, `api:"openai-completions"`. Redirect via `PI_CONFIG_DIR` (already allowlisted per CLAUDE.md) pointed at an isolated dir containing just this file, rather than overwriting the user's real one | Web-researched, unverified |
|
||||
| `grok` | Env vars `GROK_BASE_URL`, `XAI_API_KEY` (dummy ok for local; a real key for most cloud endpoints), `GROK_MODEL`. All three already fit inside the existing `XAI_*`/CLI-specific allowlist shape | Web-researched, unverified |
|
||||
| `deepseek` | Reuse the **existing** `DEEPSEEK_BASE_URL` + `DEEPSEEK_API_KEY` keys (already declared in `stock.ts:879-913`, already in `privilegedEnvKeys`). Model selection is murkier — CLAUDE.md notes dsh model is "a profile composition entry," not a flag/env var, so redirecting the endpoint is solid but forcing a specific model name may not fully work; document as best-effort and verify against a real profile | Web-researched, unverified, partial |
|
||||
| `omp` | Config file `~/.omp/agent/models.yml`-equivalent with a custom provider `baseUrl`. CLAUDE.md notes omp's config tree is itself relocatable via `PI_CONFIG_DIR` — reuse the same isolated-dir-redirect approach as `pi` | Web-researched, unverified |
|
||||
| `antigravity` | No CLI/env/config mechanism found — Antigravity's docs describe only a GUI settings panel, and explicitly say a custom endpoint "cannot currently" become the core reasoning model. **Not implemented**; toolbar entry stays disabled for this mode with an explanatory tooltip | No known mechanism |
|
||||
|
||||
Everything web-researched-but-unverified gets implemented but must be
|
||||
smoke-tested against real installs of those CLIs before being called done —
|
||||
call this out explicitly when implementing, don't just ship on faith.
|
||||
|
||||
**Cloud-endpoint specifics** to keep in mind per recipe above: an Azure AI
|
||||
Foundry-style endpoint typically wants the API key in an `api-key` header
|
||||
rather than (or in addition to) `Authorization: Bearer`, and its "model" is
|
||||
often a deployment name rather than the underlying model family name — the
|
||||
discovery step (`GET /v1/models`) still works the same way against Azure AI
|
||||
Foundry's OpenAI-compatible endpoint shape, but a user may need to type the
|
||||
deployment name manually if it isn't returned as expected.
|
||||
|
||||
## Architecture
|
||||
|
||||
### 1. Registry: new `capabilities.customModelInjection` field
|
||||
|
||||
Extend `src/config/cli-registry/types.ts` / `schema.ts` with a discriminated
|
||||
union on each `CliEntry.capabilities`:
|
||||
|
||||
```ts
|
||||
type CustomModelInjection =
|
||||
| { kind: 'env'; baseUrlVar: string; apiKeyVar: string; modelVars: string[] }
|
||||
| { kind: 'configContentEnv'; envVar: string; template: 'opencode-json' }
|
||||
| { kind: 'configDir'; dirEnvVar: string; fileName: string; template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml' }
|
||||
| { kind: 'unsupported' }
|
||||
```
|
||||
|
||||
Declared per stock.ts entry per the table above. A pure function in a new
|
||||
`src/custom-model-injection.ts` (`buildCustomModelInjection(entry, endpoint, modelId)`)
|
||||
turns `(CliEntry, endpoint, modelId)` into either an `envOverrides` object
|
||||
(kind `env`/`configContentEnv`) or a `{ dirEnvVar, files: [{path, content}] }`
|
||||
descriptor (kind `configDir`) — unit-testable with no IO, mirroring how
|
||||
`session-cli-builder.ts` is pure. The `configDir` kind additionally needs an
|
||||
IO wrapper that writes those files under
|
||||
`dataPath('custom-model-configs/<sessionId>/')` (new dir, cleaned up on
|
||||
session delete — same lifecycle as other per-session generated state).
|
||||
|
||||
### 2. Endpoint registry: `src/custom-model-hosts.ts`
|
||||
|
||||
Same read-array/write-array shape as `src/remote-hosts.ts` /
|
||||
`src/webview-store.ts`: `~/.codeman/custom-model-hosts.json` holding
|
||||
`CustomModelEndpoint[] = { id, label, baseUrl, apiKey?, authStyle?: 'bearer'|'api-key'|'both', models?: string[], lastDiscoveredAt? }`.
|
||||
`authStyle` defaults to `'both'` (send both header conventions on the
|
||||
discovery probe, same approach the smoke-test script below uses) so one
|
||||
endpoint entry works whether it's llama.cpp or Azure without the user having
|
||||
to know which header their box wants in advance.
|
||||
|
||||
New route file `src/web/routes/custom-model-routes.ts` (registered in the
|
||||
routes barrel), mirroring `case-routes.ts`'s remote/docker-host CRUD
|
||||
(`GET/POST/PUT/DELETE /api/model-endpoints`, admin-gated in multi-user mode
|
||||
the same way) plus:
|
||||
|
||||
- `POST /api/model-endpoints/:id/discover-models` — fetches
|
||||
`${baseUrl}/v1/models`, stores the `data[].id` list, returns it. Bounded
|
||||
timeout, and run the target through the **same SSRF egress guard already
|
||||
used for web tabs** (`webview-egress-policy.ts` — reject link-local/cloud
|
||||
metadata addresses) — this still matters for a cloud URL too, since the
|
||||
guard is about preventing a redirect to internal infra, not about
|
||||
local-vs-cloud.
|
||||
|
||||
**Why discovery rather than a free-text model field**: it removes the one
|
||||
piece of configuration most likely to trip a user up — hand-typing the
|
||||
exact model identifier a given inference server expects, which varies by
|
||||
server and is an easy source of a silent "model not found" failure with no
|
||||
useful error surfaced back through a CLI's own startup. Discovery also
|
||||
means this design is not limited to a single-model box: a **multi-model
|
||||
gateway** such as **[llama-swap](https://github.com/mostlygeek/llama-swap)**
|
||||
(hot-swaps between several loaded llama.cpp model configs behind one
|
||||
OpenAI-compatible endpoint) or a vLLM/LiteLLM/Ollama instance serving
|
||||
several models advertises ALL of them through the same `/v1/models` call —
|
||||
so one endpoint entry surfaces every model that gateway can serve, with no
|
||||
extra per-model configuration on Codeman's side at all.
|
||||
|
||||
### 3. Settings
|
||||
|
||||
- New synced boolean `customModelEndpointsEnabled` in `SettingsUpdateSchema`
|
||||
(`src/web/schemas.ts`), default `false`, documented inline like
|
||||
`readMyMindEnabled`/`workspaceHooksEnabled`.
|
||||
- New `.set-group` "Custom Model Endpoints" inside the **Agents & CLIs**
|
||||
section (`settings-clis`, `index.html:2150+`) with the enable toggle plus
|
||||
a list-editor (add/refresh-models/delete rows) for endpoints — closest
|
||||
existing precedent is the respawn-presets array editor
|
||||
(`schemas.ts:1285-1305`, `index.html:1243-1244`) for add/apply/delete-by-id
|
||||
semantics, backed by the new CRUD routes above.
|
||||
|
||||
### 4. Toolbar UI
|
||||
|
||||
- New header/toolbar button (e.g. `#customModelBtn`, `btn-toolbar
|
||||
btn-custom-model`), marker-hidden by default (`btn-custom-model--hidden`)
|
||||
and revealed by `applyHeaderVisibilitySettings()` only when
|
||||
`customModelEndpointsEnabled` is on — same pattern as the File
|
||||
Viewer/Cron buttons.
|
||||
- Clicking opens a dropdown (`#customModelMenu`, same `.run-mode-menu`-style
|
||||
markup as the existing Run-mode gear menu) listing "Cloud (default)" plus
|
||||
every discovered model, grouped by endpoint. An entry is disabled with a
|
||||
tooltip when the active session's CLI has `customModelInjection.kind ===
|
||||
'unsupported'` (Antigravity) or none declared.
|
||||
- Selecting an entry calls a new route:
|
||||
`POST /api/sessions/:id/custom-model { endpointId, modelId } | { clear: true }`.
|
||||
Server: resolve the CLI entry for `session.mode`, build the injection via
|
||||
§1, persist it as a new `session.customModel` state field (surfaced in
|
||||
`toState()`/SSE so the tab can show a small badge, e.g. "🖥 qwen3 (local)"
|
||||
or "☁ gpt-4o-mini (azure)", and the choice survives reload), merge into
|
||||
the session's `envOverrides`, and **respawn the pane's CLI process**
|
||||
through the same respawn/interactive-restart path
|
||||
`session.ts`/`tmux-manager.ts` already use for effort/model changes
|
||||
(`_configureCliEnv()` + `applyEnvOverrides()` at spawn time) — reuse,
|
||||
don't reinvent, the existing kill-and-relaunch-in-pane machinery.
|
||||
- New-session creation deliberately does **not** inherit a prior custom-
|
||||
endpoint choice: `buildEnvOverrides()` (session-ui.js) never carries the
|
||||
toolbar selection forward to the next `run()` call. Every new session
|
||||
starts on its native backend; picking a custom endpoint in the toolbar for
|
||||
a session applies only to that session (and, if done before Run is
|
||||
clicked, to the one session about to be created — not to sessions created
|
||||
afterward).
|
||||
|
||||
### 5. Multi-user security clamp
|
||||
|
||||
Every new env var this feature introduces that can redirect a session's
|
||||
traffic (and thus wherever its credentials go) — `ANTHROPIC_BASE_URL`,
|
||||
`GOOGLE_GEMINI_BASE_URL`, `GROK_BASE_URL`, the `CODEX_HOME`/`PI_CONFIG_DIR`
|
||||
dir-redirects, plus the already-privileged `DEEPSEEK_BASE_URL` — must be
|
||||
added to each CLI's `capabilities.privilegedEnvKeys` so
|
||||
`clampEnvOverridesForOwner()` strips them for a non-granted multi-user
|
||||
owner, exactly the precedent already documented for `DEEPSEEK_BASE_URL`/
|
||||
`OMP_AUTH_BROKER_URL`. This matters *more*, not less, now that endpoints can
|
||||
be cloud URLs: redirecting a non-granted user's session to an attacker's
|
||||
cloud endpoint is a credential-exfiltration path, not just a mischief
|
||||
redirect to a LAN box. Endpoint CRUD itself stays admin-only in multi-user
|
||||
mode, same as remote/docker hosts.
|
||||
|
||||
## Files touched (representative, not exhaustive)
|
||||
|
||||
- `src/config/cli-registry/types.ts`, `schema.ts`, `stock.ts` — new capability + per-entry declarations
|
||||
- `src/custom-model-injection.ts` (new) — pure per-CLI descriptor builder + unit tests
|
||||
- `src/custom-model-hosts.ts` (new) — endpoint store
|
||||
- `src/web/routes/custom-model-routes.ts` (new) — CRUD + discovery route
|
||||
- `src/web/routes/session-routes.ts` — `POST /api/sessions/:id/custom-model`, clamp wiring
|
||||
- `src/web/schemas.ts` — `customModelEndpointsEnabled`, endpoint/discover payload schemas, privileged-key updates
|
||||
- `src/session.ts` — `customModel` state field, `toState()` surface
|
||||
- `src/web/public/index.html`, `settings-ui.js`, `session-ui.js`, `styles.css` — settings group, toolbar button/menu, badge, accent CSS
|
||||
- `src/web/sse-events.ts` + `constants.js` — if a dedicated SSE event is warranted for the badge (or just ride existing session-update broadcasts)
|
||||
- `test/fixtures/mock-openai-server.ts` (new) + `test/custom-model-injection-contract.test.ts` (new) — see Mock-server validation below
|
||||
- `scripts/test-local-llm-harnesses.mjs` (already added, this branch) — the standalone real-CLI-and-real-endpoint smoke test; despite the filename (kept for continuity with when it was written) it already supports any `--base-url`, local or cloud
|
||||
- `docs/custom-model-endpoints.md` (new) + a CLAUDE.md pointer bullet under External CLI modes / envOverrides
|
||||
|
||||
## Mock-server validation strategy (CI-runnable, no real CLI binaries needed)
|
||||
|
||||
Spawning nine real CLI binaries in CI isn't realistic, and neither Devvyn's
|
||||
llama.cpp box nor a real cloud subscription can be a CI dependency. So the
|
||||
injection *logic* gets a tier of automated coverage that sits between the
|
||||
pure unit tests and the live manual checks in Verification:
|
||||
|
||||
1. **`test/fixtures/mock-openai-server.ts`** — a small in-process HTTP
|
||||
server (plain `http.createServer`, no external deps, port picked per the
|
||||
existing `const PORT = 3150+` convention) that:
|
||||
- Serves `GET /v1/models` → a fixed fake model list (`{data:[{id:'qwen3'},...]}`),
|
||||
for testing the discovery route.
|
||||
- Serves `POST /v1/chat/completions` (OpenAI shape) **and**
|
||||
`POST /v1/messages` (Anthropic Messages-API shape, since that's what
|
||||
`ANTHROPIC_BASE_URL` traffic looks like) and records every request it
|
||||
receives (headers, body, path) into an array the test can assert on —
|
||||
including which auth header style it saw, so the `authStyle: 'both'`
|
||||
default and Azure's `api-key` convention both get real coverage.
|
||||
- Returns a minimal valid completion so a client library doesn't choke
|
||||
on the response shape.
|
||||
|
||||
2. **`test/custom-model-injection-contract.test.ts`** — for every CLI with a
|
||||
`customModelInjection` capability (i.e. every row in the table above
|
||||
except `antigravity`):
|
||||
- Point a fixture `CustomModelEndpoint` at the mock server's URL.
|
||||
- Call `buildCustomModelInjection(entry, endpoint, modelId)` (the pure
|
||||
function from §1) to get the real env vars / config-file content that
|
||||
would be injected into that CLI's session.
|
||||
- Replay those exact values through a minimal HTTP request shaped the
|
||||
way that CLI is documented to send it (Anthropic Messages shape for
|
||||
claude; OpenAI chat-completions shape for opencode/codex/pi/grok/omp;
|
||||
`GOOGLE_GEMINI_BASE_URL`'s OpenAI-compat shape for gemini; dsh's
|
||||
provider call for deepseek) against the mock server.
|
||||
- Assert the mock server received the request **at the injected
|
||||
`baseUrl`**, with **the injected API key** in the expected header, and
|
||||
**the injected model id** in the body/path — i.e. prove the values
|
||||
Codeman computes are internally consistent and would reach the right
|
||||
place with the right identifiers, end to end, in CI, on every push.
|
||||
- Also cover the `configDir` kind (codex/pi/omp): assert the written
|
||||
`config.toml`/`models.json`/`models.yml` file parses and contains the
|
||||
same base URL/key/model, and that it's written under the isolated
|
||||
per-session dir rather than the user's real config path.
|
||||
|
||||
3. **Explicit, stated limitation** (goes in the test file's `@fileoverview`
|
||||
and in this doc, not left implicit): this proves *"if the CLI honors its
|
||||
documented env/config contract, it will hit the right endpoint with the
|
||||
right model."* It does **not** prove the real CLI binary actually reads
|
||||
that env var / config file the way its docs say — that's still the job
|
||||
of the live manual checks in Verification step 4-5 below, and is exactly
|
||||
why the confidence table above stays "unverified" for six of the nine
|
||||
CLIs until someone runs those binaries for real. The mock-server suite
|
||||
catches regressions in Codeman's own logic; it cannot catch a CLI
|
||||
changing its env-var name in a future release, or a real cloud endpoint
|
||||
behaving differently from the mock.
|
||||
|
||||
## Verification
|
||||
|
||||
1. `npm run typecheck && npm test` after each slice — this now includes the
|
||||
mock-server contract suite from above, so injection-logic regressions
|
||||
are caught automatically without touching real infrastructure.
|
||||
2. Unit tests for `buildCustomModelInjection()` per CLI kind (pure, no IO).
|
||||
3. Route tests (`app.inject`) for the new CRUD + discover-models endpoint
|
||||
(mock `fetch` for `/v1/models`), and for the multi-user clamp on the new
|
||||
privileged keys (mirror `test/routes/external-cli-bypass-clamp.test.ts`).
|
||||
4. **Standalone real-binary smoke test**: `scripts/test-local-llm-harnesses.mjs`
|
||||
(already written on this branch) exercises every harness against a real
|
||||
`--base-url` — local or cloud — outside of Codeman's UI entirely. Run it
|
||||
against Devvyn's llama.cpp server first (`claude`/`opencode`/`codex`
|
||||
should PASS, since those recipes are verified; the rest report
|
||||
UNCONFIRMED/SKIP until their guessed flags are corrected via
|
||||
`--probe-help`), then again against a real cloud endpoint (e.g. an Azure
|
||||
AI Foundry deployment) once one is available, to prove the `authStyle`/
|
||||
deployment-name handling holds up outside llama.cpp.
|
||||
5. Once the full feature (not just the standalone script) is built: add an
|
||||
endpoint via the real UI, hit discover-models, confirm the returned model
|
||||
list, pick Claude + the model on a real session, confirm via
|
||||
`tmux -L codeman capture-pane`/`tmux showenv -t <pane>` that
|
||||
`ANTHROPIC_BASE_URL`/`ANTHROPIC_API_KEY`/`ANTHROPIC_DEFAULT_*_MODEL` are
|
||||
set post-restart, and confirm the endpoint's own logs show the next
|
||||
prompt actually landing there. Repeat for opencode and Codex at minimum
|
||||
before considering this shippable; spot-check the web-researched CLIs
|
||||
and correct the plan's confidence table with what's actually observed.
|
||||
6. `npm run lint && npm run format:check`.
|
||||
7. Update `CHANGELOG.md`/changeset per the COM workflow when shipping.
|
||||
Reference in New Issue
Block a user