test(custom-model): make the harness smoke test dynamic, verify all 9 CLIs end-to-end

Rewrites scripts/test-local-llm-harnesses.mjs -> .ts to read the live CLI
registry (enabledClis()) and call the real production
buildCustomModelInjection()/applyConfigDirInjection() instead of keeping a
second hand-maintained copy of every CLI's env/config shape. A future
registry change (new CLI, edited env var, fixed config template) is now
picked up automatically with zero edits to this script; only the one-shot
invocation flags (info the registry genuinely doesn't model) stay in a
small hand-maintained ONE_SHOT table, and a registry CLI with no entry
there reports UNKNOWN rather than being silently skipped.

Extracted src/custom-model-injection-apply.ts (applyConfigDirInjection/
removeConfigDir) so the production route and this script share one
implementation instead of two.

Full end-to-end run against a real llama-swap server, inside a
codeman/agent:llm-test Docker image with all 9 CLI binaries:

- claude, opencode, pi, grok, omp: PASS, real "hello world" replies
- codex: confirmed FAIL for a real protocol reason, not a bug — it only
  speaks the Responses API since Feb 2026, which llama.cpp/llama-swap
  don't implement
- gemini: confirmed FAIL, unresolved after real investigation — an
  undocumented GATEWAY AuthType gemini-cli selects once
  GOOGLE_GEMINI_BASE_URL is set rejects every auth-key format/override
  tried
- deepseek: reaches the server (env vars are read) but gets a consistent
  HTTP_404; root cause not identified, documented as best-effort/unknown
- antigravity: SKIP, no known mechanism (unchanged)

Two real bugs found and fixed along the way (grok, pi/omp registry
entries in stock.ts): grok's original recipe (env vars) was flat-out
wrong, not just unverified — the real mechanism is a config.toml
[model.<name>] block redirected via GROK_HOME. pi/omp's PI_CONFIG_DIR
does nothing for either (grepped pi's entire bundled source — the string
appears nowhere); the real redirect is the child process's own HOME, and
both need `models` as an array of {id} objects, not an object keyed by
id (silently loaded zero models otherwise).

deployment_plan.md, PR.md, docs/custom-model-endpoints.md, and CLAUDE.md
updated with the final confidence table reflecting all of the above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HqNWfmtBU2KN29SvSVWB3

src/config/cli-registry/schema.ts

@@ -333,7 +333,7 @@ const capabilitiesSchema = z
           kind: z.literal('configDir'),
           dirEnvVar: envName,
           fileName: z.string().min(1).max(80),
-          template: z.enum(['codex-toml', 'pi-models-json', 'omp-models-yml']),
+          template: z.enum(['codex-toml', 'pi-models-json', 'omp-models-yml', 'grok-toml']),
         })
         .strict(),
       z.object({ kind: z.literal('unsupported') }).strict(),

src/config/cli-registry/stock.ts

@@ -749,20 +749,30 @@ const PI: CliEntry = {
     // just answer "yes" to, so omitting --approve is not itself a clamp — MATERIALIZE
     // approveProjectTrust:false so buildPiCommand emits --no-approve outright.
     privilegedParams: [{ param: 'approveProjectTrust', clampTo: false, materializeWhenAbsent: true }],
-    // Web-researched, unverified. pi's models.json hot-reloads, but this feature always
-    // restarts the CLI on switch for consistency with the other 8 harnesses. Written to an
-    // isolated PI_CONFIG_DIR so the user's real ~/.pi/agent/models.json is never touched.
+    // CORRECTED after live-testing: `PI_CONFIG_DIR` does NOT exist anywhere in pi's own
+    // bundled source (grepped the installed package directly) — it does nothing for pi
+    // itself, despite being a real Codeman env var that OTHER things (omp) read. The
+    // confirmed working redirect is `HOME` itself: pi hardcodes `~/.pi/agent/models.json`
+    // with no dedicated override, so redirecting the CHILD PROCESS's HOME is what
+    // actually relocates it (verified: a model written under an isolated HOME's
+    // `.pi/agent/models.json` shows up in `pi --list-models` and answers a real prompt
+    // against a real llama-swap server; PI_CONFIG_DIR alone left it silently unable to
+    // see any provider). ⚠️ This is a bigger blast radius than a dedicated config-dir
+    // var: it also redirects pi's real sessions/auth/extensions for the DURATION of a
+    // custom-model session, not just its provider config — document this trade-off
+    // wherever this capability is surfaced.
     customModelInjection: {
       kind: 'configDir',
-      dirEnvVar: 'PI_CONFIG_DIR',
-      fileName: 'agent/models.json',
+      dirEnvVar: 'HOME',
+      fileName: '.pi/agent/models.json',
       template: 'pi-models-json',
     },
-    // PI_CONFIG_DIR already matches the PI_ allowedPrefix above, so it was ALREADY
-    // reachable via plain envOverrides before this feature existed — and pi executes
-    // repo-local .pi/extensions TypeScript (see the External CLI modes note in CLAUDE.md),
-    // so redirecting this dir is a code-execution surface, not just a config swap.
-    privilegedEnvKeys: ['PI_CONFIG_DIR'],
+    // HOME is not `PI_`-prefixed, so unlike the old (wrong) PI_CONFIG_DIR guess this was
+    // never reachable via the generic envOverrides allowlist at all — listed here anyway,
+    // matching the documented pattern for every other CLI's dir-redirect var, since a
+    // redirected HOME is at least as sensitive as CODEX_HOME/GROK_HOME (pi executes
+    // repo-local .pi/extensions TypeScript — see the External CLI modes note in CLAUDE.md).
+    privilegedEnvKeys: ['HOME'],
   },
   overlays: {
     credStore: {
@@ -858,16 +868,24 @@ const GROK: CliEntry = {
     // already its safe interactive ask-mode, so the multi-user clamp only needs to force an
     // EXPLICITLY-SENT bypass flag back off — nothing is materialized when config is absent.
     privilegedParams: [{ param: 'alwaysApprove', clampTo: false }],
-    // Web-researched, unverified.
+    // CORRECTED after live-testing against a real grok binary: the original `env` kind
+    // (GROK_BASE_URL/GROK_MODEL/XAI_API_KEY) produced "Not signed in" — those env vars
+    // are NOT grok's real custom-endpoint mechanism. The real one (verified against
+    // xAI's own docs) is a `[model.<name>]` block in a config.toml under GROK_HOME,
+    // the same configDir shape as codex/pi/omp. `api_backend = "chat_completions"` is
+    // explicitly supported (unlike codex, which dropped it) — grok CAN talk to a plain
+    // OpenAI Chat-Completions server directly.
     customModelInjection: {
-      kind: 'env',
-      baseUrlVar: 'GROK_BASE_URL',
-      apiKeyVar: 'XAI_API_KEY',
-      modelVars: ['GROK_MODEL'],
+      kind: 'configDir',
+      dirEnvVar: 'GROK_HOME',
+      fileName: 'config.toml',
+      template: 'grok-toml',
     },
-    // All three already match the GROK_/XAI_ allowedPrefixes above, so they were ALREADY
-    // reachable via plain envOverrides before this feature existed.
-    privilegedEnvKeys: ['GROK_BASE_URL', 'XAI_API_KEY', 'GROK_MODEL'],
+    // GROK_HOME already matches the GROK_ allowedPrefix above, so it was ALREADY
+    // reachable via plain envOverrides before this feature existed — same reasoning
+    // as CODEX_HOME: a redirected config dir can restate policy the argv-level
+    // `alwaysApprove` clamp above cannot see.
+    privilegedEnvKeys: ['GROK_HOME'],
   },
   overlays: {
     // ~/.grok also holds sessions/, memory/, downloads/ (the ~160MB binary), completions/,
@@ -1133,18 +1151,20 @@ const OMP: CliEntry = {
     // Where omp resolves its auth from. No known concrete exfiltration path today (omp
     // forwards no operator-held key into a pane), but a non-granted owner redirecting where
     // a shared multi-tenant deployment resolves auth is not something to allow silently.
-    // PI_CONFIG_DIR added for custom-model-injection.ts's omp recipe, which reuses pi's
-    // dir-redirect mechanism (see the customModelInjection comment below) — already
-    // reachable via the PI_ allowedPrefix (pi's own entry), so this closes the same
-    // pre-existing gap for an omp session that PI's own entry closes for a pi session.
-    privilegedEnvKeys: ['OMP_AUTH_BROKER_URL', 'OMP_AUTH_BROKER_TOKEN', 'PI_CONFIG_DIR'],
-    // Web-researched, unverified. omp's ~/.omp tree is itself relocatable via PI_CONFIG_DIR
-    // (see the DeepSeek/OMP note in CLAUDE.md), so this reuses that same redirect rather
-    // than inventing an OMP-specific dir env var.
+    // HOME added for custom-model-injection.ts's omp recipe (see below). Unlike pi,
+    // PI_CONFIG_DIR genuinely IS one of the env vars omp reads (per the DeepSeek/OMP
+    // note in CLAUDE.md) — but live-testing this feature found it did NOT relocate
+    // omp's model config the way expected, while redirecting HOME itself (like pi)
+    // worked immediately (verified end-to-end: a real "hello world" reply came back).
+    privilegedEnvKeys: ['OMP_AUTH_BROKER_URL', 'OMP_AUTH_BROKER_TOKEN', 'HOME'],
+    // Verified end-to-end against a real llama-swap server (live-tested, not just
+    // researched — a real "hello world" reply came back). Same HOME-redirect mechanism
+    // as pi (see its customModelInjection comment for the full reasoning) — omp hardcodes
+    // `~/.omp/agent/models.yml` with no dedicated config-dir override either.
     customModelInjection: {
       kind: 'configDir',
-      dirEnvVar: 'PI_CONFIG_DIR',
-      fileName: 'agent/models.yml',
+      dirEnvVar: 'HOME',
+      fileName: '.omp/agent/models.yml',
       template: 'omp-models-yml',
     },
   },

src/config/cli-registry/types.ts

@@ -452,9 +452,18 @@ export interface CliCapabilities {
    * carried in one env var (opencode's `OPENCODE_CONFIG_CONTENT`).
    * `configDir`: a generated config file under an isolated, dir-redirect-env-
    * pointed directory so the user's real CLI config is never touched
-   * (codex's `CODEX_HOME`/`config.toml`, pi/omp's `PI_CONFIG_DIR`).
-   * `unsupported`: no known mechanism (antigravity) — the toolbar entry
-   * stays disabled for this CLI.
+   * (codex's `CODEX_HOME`/`config.toml`, pi/omp's `PI_CONFIG_DIR`, grok's
+   * `GROK_HOME`/`config.toml`). `unsupported`: no known mechanism
+   * (antigravity) — the toolbar entry stays disabled for this CLI.
+   *
+   * ⚠️ grok was ORIGINALLY declared as `env` kind (`GROK_BASE_URL`/
+   * `GROK_MODEL`/`XAI_API_KEY`) — that recipe was WRONG, not just unverified:
+   * live-tested against a real grok binary, it produced "Not signed in",
+   * because those env vars are not grok's real custom-endpoint mechanism at
+   * all. The real one is a `[model.<name>]` block in a `config.toml` under
+   * `GROK_HOME` (verified against xAI's own docs), same shape as codex/pi/
+   * omp — this is why the confidence table in deployment_plan.md exists:
+   * "researched" web docs can still be plausible-sounding and wrong.
    *
    * Every env var name this introduces that can redirect a session's
    * traffic MUST also appear in `privilegedEnvKeys` above, exactly like
@@ -469,7 +478,7 @@ export interface CliCapabilities {
         kind: 'configDir';
         dirEnvVar: string;
         fileName: string;
-        template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml';
+        template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml' | 'grok-toml';
       }
     | { kind: 'unsupported' };
 }