test(custom-model): make the harness smoke test dynamic, verify all 9 CLIs end-to-end

Rewrites scripts/test-local-llm-harnesses.mjs -> .ts to read the live CLI
registry (enabledClis()) and call the real production
buildCustomModelInjection()/applyConfigDirInjection() instead of keeping a
second hand-maintained copy of every CLI's env/config shape. A future
registry change (new CLI, edited env var, fixed config template) is now
picked up automatically with zero edits to this script; only the one-shot
invocation flags (info the registry genuinely doesn't model) stay in a
small hand-maintained ONE_SHOT table, and a registry CLI with no entry
there reports UNKNOWN rather than being silently skipped.

Extracted src/custom-model-injection-apply.ts (applyConfigDirInjection/
removeConfigDir) so the production route and this script share one
implementation instead of two.

Full end-to-end run against a real llama-swap server, inside a
codeman/agent:llm-test Docker image with all 9 CLI binaries:

- claude, opencode, pi, grok, omp: PASS, real "hello world" replies
- codex: confirmed FAIL for a real protocol reason, not a bug — it only
  speaks the Responses API since Feb 2026, which llama.cpp/llama-swap
  don't implement
- gemini: confirmed FAIL, unresolved after real investigation — an
  undocumented GATEWAY AuthType gemini-cli selects once
  GOOGLE_GEMINI_BASE_URL is set rejects every auth-key format/override
  tried
- deepseek: reaches the server (env vars are read) but gets a consistent
  HTTP_404; root cause not identified, documented as best-effort/unknown
- antigravity: SKIP, no known mechanism (unchanged)

Two real bugs found and fixed along the way (grok, pi/omp registry
entries in stock.ts): grok's original recipe (env vars) was flat-out
wrong, not just unverified — the real mechanism is a config.toml
[model.<name>] block redirected via GROK_HOME. pi/omp's PI_CONFIG_DIR
does nothing for either (grepped pi's entire bundled source — the string
appears nowhere); the real redirect is the child process's own HOME, and
both need `models` as an array of {id} objects, not an object keyed by
id (silently loaded zero models otherwise).

deployment_plan.md, PR.md, docs/custom-model-endpoints.md, and CLAUDE.md
updated with the final confidence table reflecting all of the above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HqNWfmtBU2KN29SvSVWB3
This commit is contained in:
Devvyn
2026-09-13 17:42:35 +08:00
co-authored by Claude Sonnet 5
parent 41416566aa
commit 61779745aa
14 changed files with 1061 additions and 770 deletions
+1 -1
View File
@@ -333,7 +333,7 @@ const capabilitiesSchema = z
kind: z.literal('configDir'),
dirEnvVar: envName,
fileName: z.string().min(1).max(80),
template: z.enum(['codex-toml', 'pi-models-json', 'omp-models-yml']),
template: z.enum(['codex-toml', 'pi-models-json', 'omp-models-yml', 'grok-toml']),
})
.strict(),
z.object({ kind: z.literal('unsupported') }).strict(),
+48 -28
View File
@@ -749,20 +749,30 @@ const PI: CliEntry = {
// just answer "yes" to, so omitting --approve is not itself a clamp — MATERIALIZE
// approveProjectTrust:false so buildPiCommand emits --no-approve outright.
privilegedParams: [{ param: 'approveProjectTrust', clampTo: false, materializeWhenAbsent: true }],
// Web-researched, unverified. pi's models.json hot-reloads, but this feature always
// restarts the CLI on switch for consistency with the other 8 harnesses. Written to an
// isolated PI_CONFIG_DIR so the user's real ~/.pi/agent/models.json is never touched.
// CORRECTED after live-testing: `PI_CONFIG_DIR` does NOT exist anywhere in pi's own
// bundled source (grepped the installed package directly) — it does nothing for pi
// itself, despite being a real Codeman env var that OTHER things (omp) read. The
// confirmed working redirect is `HOME` itself: pi hardcodes `~/.pi/agent/models.json`
// with no dedicated override, so redirecting the CHILD PROCESS's HOME is what
// actually relocates it (verified: a model written under an isolated HOME's
// `.pi/agent/models.json` shows up in `pi --list-models` and answers a real prompt
// against a real llama-swap server; PI_CONFIG_DIR alone left it silently unable to
// see any provider). ⚠️ This is a bigger blast radius than a dedicated config-dir
// var: it also redirects pi's real sessions/auth/extensions for the DURATION of a
// custom-model session, not just its provider config — document this trade-off
// wherever this capability is surfaced.
customModelInjection: {
kind: 'configDir',
dirEnvVar: 'PI_CONFIG_DIR',
fileName: 'agent/models.json',
dirEnvVar: 'HOME',
fileName: '.pi/agent/models.json',
template: 'pi-models-json',
},
// PI_CONFIG_DIR already matches the PI_ allowedPrefix above, so it was ALREADY
// reachable via plain envOverrides before this feature existed — and pi executes
// repo-local .pi/extensions TypeScript (see the External CLI modes note in CLAUDE.md),
// so redirecting this dir is a code-execution surface, not just a config swap.
privilegedEnvKeys: ['PI_CONFIG_DIR'],
// HOME is not `PI_`-prefixed, so unlike the old (wrong) PI_CONFIG_DIR guess this was
// never reachable via the generic envOverrides allowlist at all — listed here anyway,
// matching the documented pattern for every other CLI's dir-redirect var, since a
// redirected HOME is at least as sensitive as CODEX_HOME/GROK_HOME (pi executes
// repo-local .pi/extensions TypeScript — see the External CLI modes note in CLAUDE.md).
privilegedEnvKeys: ['HOME'],
},
overlays: {
credStore: {
@@ -858,16 +868,24 @@ const GROK: CliEntry = {
// already its safe interactive ask-mode, so the multi-user clamp only needs to force an
// EXPLICITLY-SENT bypass flag back off — nothing is materialized when config is absent.
privilegedParams: [{ param: 'alwaysApprove', clampTo: false }],
// Web-researched, unverified.
// CORRECTED after live-testing against a real grok binary: the original `env` kind
// (GROK_BASE_URL/GROK_MODEL/XAI_API_KEY) produced "Not signed in" — those env vars
// are NOT grok's real custom-endpoint mechanism. The real one (verified against
// xAI's own docs) is a `[model.<name>]` block in a config.toml under GROK_HOME,
// the same configDir shape as codex/pi/omp. `api_backend = "chat_completions"` is
// explicitly supported (unlike codex, which dropped it) — grok CAN talk to a plain
// OpenAI Chat-Completions server directly.
customModelInjection: {
kind: 'env',
baseUrlVar: 'GROK_BASE_URL',
apiKeyVar: 'XAI_API_KEY',
modelVars: ['GROK_MODEL'],
kind: 'configDir',
dirEnvVar: 'GROK_HOME',
fileName: 'config.toml',
template: 'grok-toml',
},
// All three already match the GROK_/XAI_ allowedPrefixes above, so they were ALREADY
// reachable via plain envOverrides before this feature existed.
privilegedEnvKeys: ['GROK_BASE_URL', 'XAI_API_KEY', 'GROK_MODEL'],
// GROK_HOME already matches the GROK_ allowedPrefix above, so it was ALREADY
// reachable via plain envOverrides before this feature existed — same reasoning
// as CODEX_HOME: a redirected config dir can restate policy the argv-level
// `alwaysApprove` clamp above cannot see.
privilegedEnvKeys: ['GROK_HOME'],
},
overlays: {
// ~/.grok also holds sessions/, memory/, downloads/ (the ~160MB binary), completions/,
@@ -1133,18 +1151,20 @@ const OMP: CliEntry = {
// Where omp resolves its auth from. No known concrete exfiltration path today (omp
// forwards no operator-held key into a pane), but a non-granted owner redirecting where
// a shared multi-tenant deployment resolves auth is not something to allow silently.
// PI_CONFIG_DIR added for custom-model-injection.ts's omp recipe, which reuses pi's
// dir-redirect mechanism (see the customModelInjection comment below) — already
// reachable via the PI_ allowedPrefix (pi's own entry), so this closes the same
// pre-existing gap for an omp session that PI's own entry closes for a pi session.
privilegedEnvKeys: ['OMP_AUTH_BROKER_URL', 'OMP_AUTH_BROKER_TOKEN', 'PI_CONFIG_DIR'],
// Web-researched, unverified. omp's ~/.omp tree is itself relocatable via PI_CONFIG_DIR
// (see the DeepSeek/OMP note in CLAUDE.md), so this reuses that same redirect rather
// than inventing an OMP-specific dir env var.
// HOME added for custom-model-injection.ts's omp recipe (see below). Unlike pi,
// PI_CONFIG_DIR genuinely IS one of the env vars omp reads (per the DeepSeek/OMP
// note in CLAUDE.md) — but live-testing this feature found it did NOT relocate
// omp's model config the way expected, while redirecting HOME itself (like pi)
// worked immediately (verified end-to-end: a real "hello world" reply came back).
privilegedEnvKeys: ['OMP_AUTH_BROKER_URL', 'OMP_AUTH_BROKER_TOKEN', 'HOME'],
// Verified end-to-end against a real llama-swap server (live-tested, not just
// researched — a real "hello world" reply came back). Same HOME-redirect mechanism
// as pi (see its customModelInjection comment for the full reasoning) — omp hardcodes
// `~/.omp/agent/models.yml` with no dedicated config-dir override either.
customModelInjection: {
kind: 'configDir',
dirEnvVar: 'PI_CONFIG_DIR',
fileName: 'agent/models.yml',
dirEnvVar: 'HOME',
fileName: '.omp/agent/models.yml',
template: 'omp-models-yml',
},
},
+13 -4
View File
@@ -452,9 +452,18 @@ export interface CliCapabilities {
* carried in one env var (opencode's `OPENCODE_CONFIG_CONTENT`).
* `configDir`: a generated config file under an isolated, dir-redirect-env-
* pointed directory so the user's real CLI config is never touched
* (codex's `CODEX_HOME`/`config.toml`, pi/omp's `PI_CONFIG_DIR`).
* `unsupported`: no known mechanism (antigravity) — the toolbar entry
* stays disabled for this CLI.
* (codex's `CODEX_HOME`/`config.toml`, pi/omp's `PI_CONFIG_DIR`, grok's
* `GROK_HOME`/`config.toml`). `unsupported`: no known mechanism
* (antigravity) — the toolbar entry stays disabled for this CLI.
*
* ⚠️ grok was ORIGINALLY declared as `env` kind (`GROK_BASE_URL`/
* `GROK_MODEL`/`XAI_API_KEY`) — that recipe was WRONG, not just unverified:
* live-tested against a real grok binary, it produced "Not signed in",
* because those env vars are not grok's real custom-endpoint mechanism at
* all. The real one is a `[model.<name>]` block in a `config.toml` under
* `GROK_HOME` (verified against xAI's own docs), same shape as codex/pi/
* omp — this is why the confidence table in deployment_plan.md exists:
* "researched" web docs can still be plausible-sounding and wrong.
*
* Every env var name this introduces that can redirect a session's
* traffic MUST also appear in `privilegedEnvKeys` above, exactly like
@@ -469,7 +478,7 @@ export interface CliCapabilities {
kind: 'configDir';
dirEnvVar: string;
fileName: string;
template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml';
template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml' | 'grok-toml';
}
| { kind: 'unsupported' };
}
+42
View File
@@ -0,0 +1,42 @@
/**
* @fileoverview The one IO wrapper around `custom-model-injection.ts`'s pure
* `ConfigDirInjection` output — deliberately split out so that file, the
* discovery routes, and `scripts/test-local-llm-harnesses.ts` (via tsx) can
* all share EXACTLY one "write these files, merge this env" implementation.
* Before this existed, the route and the standalone script each carried
* their own copy of this logic, which is exactly the kind of drift the CLI
* registry's "declare once, consume everywhere" design exists to prevent —
* see deployment_plan.md and the "dynamic to support cli-registry changes"
* requirement it was written against.
*/
import { mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { join, dirname } from 'node:path';
import type { ConfigDirInjection } from './custom-model-injection.js';
/**
* Writes a `ConfigDirInjection`'s files under `baseDir` and returns the full
* envOverrides object a caller should merge into the session/process env
* (the dir-redirect var plus any `extraEnv` the config file references by
* name). Never touches anything outside `baseDir` — the caller is
* responsible for choosing an isolated directory (never the user's real
* `~/.codex`, `~/.pi`, etc.).
*/
export function applyConfigDirInjection(baseDir: string, injection: ConfigDirInjection): Record<string, string> {
for (const file of injection.files) {
const filePath = join(baseDir, file.relPath);
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, file.content, 'utf8');
}
return { [injection.dirEnvVar]: baseDir, ...injection.extraEnv };
}
/** Best-effort recursive removal of a previously-written configDir. Never throws. */
export function removeConfigDir(dir: string | undefined): void {
if (!dir) return;
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// best-effort cleanup only
}
}
+46 -5
View File
@@ -14,7 +14,7 @@
* llama-swap server (a real "hello world" reply came back). `codex`'s
* config.toml STRUCTURE is now verified (an earlier `[model].default` table
* shape was rejected by a real codex binary with "invalid type: map,
* expected a string" — caught by `scripts/test-local-llm-harnesses.mjs`),
* expected a string" — caught by `scripts/test-local-llm-harnesses.ts`),
* but `wire_api = "responses"` is the only value codex still accepts
* (support for `"chat"` was dropped in Feb 2026), and a plain OpenAI
* Chat-Completions server (llama.cpp, llama-swap, most local setups) does
@@ -134,8 +134,12 @@ function renderConfigContent(
const CODEX_API_KEY_ENV_VAR = 'CODEMAN_CUSTOM_MODEL_API_KEY';
/** The `[model.<name>]` block name grok's config.toml uses for the injected model — also
* what `-m <name>` in the standalone script's ONE_SHOT argv must reference to select it. */
export const GROK_CUSTOM_MODEL_NAME = 'codeman-custom';
function renderConfigFile(
template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml',
template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml' | 'grok-toml',
endpoint: CustomModelEndpoint,
modelId: string,
apiKey: string
@@ -145,7 +149,7 @@ function renderConfigFile(
case 'codex-toml': {
// Verified against real codex (>= Feb 2026): `model` is a top-level STRING, never
// a `[model].default` table — codex rejects that with "invalid type: map, expected
// a string" (caught by scripts/test-local-llm-harnesses.mjs against a real llama-swap
// a string" (caught by scripts/test-local-llm-harnesses.ts against a real llama-swap
// server). The API key is NEVER a literal TOML field: codex's schema only supports
// `env_key`, the NAME of an env var it reads the credential from at runtime, so the
// actual value must ride along as an extra env var, never embedded in the file.
@@ -169,11 +173,23 @@ function renderConfigFile(
return { content, extraEnv: { [CODEX_API_KEY_ENV_VAR]: apiKey } };
}
case 'pi-models-json':
// Verified against pi's OWN bundled docs (models.md): `models` is an ARRAY of
// `{id: "..."}` objects, NOT an object keyed by model id — the earlier shape here
// silently loaded zero models ("No models available"), confirmed live. `authHeader:
// true` is required too: pi does not automatically send `Authorization: Bearer
// <apiKey>` just because `apiKey` is set (per the same doc) — without it, a real
// (non-llama.cpp) endpoint that actually checks the key would reject every request.
return {
content: JSON.stringify(
{
providers: {
custom: { baseUrl, apiKey, api: 'openai-completions', models: { [modelId]: {} } },
custom: {
baseUrl,
apiKey,
api: 'openai-completions',
authHeader: true,
models: [{ id: modelId }],
},
},
},
null,
@@ -181,8 +197,33 @@ function renderConfigFile(
),
};
case 'omp-models-yml':
// Mirrors the pi-models-json fix above (omp shares pi's config lineage per
// CLAUDE.md — it reads several of pi's own env vars): a flat list of bare model
// name strings under `models` is UNCONFIRMED against real omp docs (none are
// bundled with the binary) — this now matches pi's `{id: "..."}` object-list
// shape and adds `authHeader: true` on the same reasoning, but has not itself
// been live-tested the way pi's fix was. Verify before raising its confidence.
return {
content: `providers:\n custom:\n baseUrl: ${quoted(baseUrl)}\n apiKey: ${quoted(apiKey)}\n models:\n - ${quoted(modelId)}\n`,
content: `providers:\n custom:\n baseUrl: ${quoted(baseUrl)}\n apiKey: ${quoted(apiKey)}\n api: openai-completions\n authHeader: true\n models:\n - id: ${quoted(modelId)}\n`,
};
case 'grok-toml': {
// Verified against xAI's own docs (docs.x.ai/build/settings/reference): a
// `[model.<name>]` block, NOT plain env vars — an earlier `env`-kind recipe for
// grok was wrong, not just unverified (see the customModelInjection doc comment
// in cli-registry/types.ts). `api_backend = "chat_completions"` is explicitly
// supported (unlike codex, which dropped it after Feb 2026), so this one CAN
// talk to a plain OpenAI-compatible server directly. `env_key` reuses grok's own
// documented fallback var name (XAI_API_KEY) rather than inventing a new one.
const content = [
`[model.${GROK_CUSTOM_MODEL_NAME}]`,
`model = ${quoted(modelId)}`,
`base_url = ${quoted(baseUrl)}`,
`name = "Custom Endpoint"`,
`env_key = "XAI_API_KEY"`,
`api_backend = "chat_completions"`,
'',
].join('\n');
return { content, extraEnv: { XAI_API_KEY: apiKey } };
}
}
}
+6 -13
View File
@@ -8,7 +8,7 @@ import { FastifyInstance, type FastifyReply } from 'fastify';
import { z } from 'zod';
import { join, dirname, extname, basename } from 'node:path';
import { homedir } from 'node:os';
import { existsSync, statSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { existsSync, statSync, mkdirSync, writeFileSync } from 'node:fs';
import { execFile } from 'node:child_process';
import fs from 'node:fs/promises';
import { randomBytes } from 'node:crypto';
@@ -55,6 +55,7 @@ import {
} from '../schemas.js';
import { readCustomModelHosts } from '../../custom-model-hosts.js';
import { buildCustomModelInjection } from '../../custom-model-injection.js';
import { applyConfigDirInjection, removeConfigDir } from '../../custom-model-injection-apply.js';
import { ownerLayoutKey } from '../../tab-layout-persistence.js';
import { TabLayoutValidationError } from '../../tab-layout.js';
import {
@@ -1184,7 +1185,7 @@ export function registerSessionRoutes(
if ('clear' in body) {
const previousConfigDir = session.setCustomModel(undefined);
if (previousConfigDir) rmSync(previousConfigDir, { recursive: true, force: true });
removeConfigDir(previousConfigDir);
const restarted = await session.restartCli();
persistAndBroadcastSession(ctx, session);
return { customModel: session.customModel, restarted };
@@ -1216,16 +1217,8 @@ export function registerSessionRoutes(
} else if (injection.kind === 'configDir') {
// Isolated per-session dir — never the user's real CLI config path.
configDir = join(dataPath('custom-model-configs'), session.id);
for (const file of injection.files) {
const filePath = join(configDir, file.relPath);
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, file.content, 'utf8');
}
// extraEnv: vars the written config file REFERENCES by name (codex's `env_key`
// convention) rather than embedding a literal value — must ride alongside
// dirEnvVar or the config points at a credential that was never actually set.
envOverrides = { [injection.dirEnvVar]: configDir, ...injection.extraEnv };
envKeys = [injection.dirEnvVar, ...Object.keys(injection.extraEnv ?? {})];
envOverrides = applyConfigDirInjection(configDir, injection);
envKeys = Object.keys(envOverrides);
} else {
// 'unsupported' is already handled above; this keeps the switch exhaustive.
return createErrorResponse(ApiErrorCode.OPERATION_FAILED, `${session.mode} has no known custom-model mechanism`);
@@ -1238,7 +1231,7 @@ export function registerSessionRoutes(
// Clean up the OLD config dir on disk, unless the new one happens to reuse the same
// path (same session, configDir kind again) — never delete the dir we just wrote.
if (previousConfigDir && previousConfigDir !== configDir) {
rmSync(previousConfigDir, { recursive: true, force: true });
removeConfigDir(previousConfigDir);
}
const restarted = await session.restartCli();