test(custom-model): make the harness smoke test dynamic, verify all 9 CLIs end-to-end

Rewrites scripts/test-local-llm-harnesses.mjs -> .ts to read the live CLI
registry (enabledClis()) and call the real production
buildCustomModelInjection()/applyConfigDirInjection() instead of keeping a
second hand-maintained copy of every CLI's env/config shape. A future
registry change (new CLI, edited env var, fixed config template) is now
picked up automatically with zero edits to this script; only the one-shot
invocation flags (info the registry genuinely doesn't model) stay in a
small hand-maintained ONE_SHOT table, and a registry CLI with no entry
there reports UNKNOWN rather than being silently skipped.

Extracted src/custom-model-injection-apply.ts (applyConfigDirInjection/
removeConfigDir) so the production route and this script share one
implementation instead of two.

Full end-to-end run against a real llama-swap server, inside a
codeman/agent:llm-test Docker image with all 9 CLI binaries:

- claude, opencode, pi, grok, omp: PASS, real "hello world" replies
- codex: confirmed FAIL for a real protocol reason, not a bug — it only
  speaks the Responses API since Feb 2026, which llama.cpp/llama-swap
  don't implement
- gemini: confirmed FAIL, unresolved after real investigation — an
  undocumented GATEWAY AuthType gemini-cli selects once
  GOOGLE_GEMINI_BASE_URL is set rejects every auth-key format/override
  tried
- deepseek: reaches the server (env vars are read) but gets a consistent
  HTTP_404; root cause not identified, documented as best-effort/unknown
- antigravity: SKIP, no known mechanism (unchanged)

Two real bugs found and fixed along the way (grok, pi/omp registry
entries in stock.ts): grok's original recipe (env vars) was flat-out
wrong, not just unverified — the real mechanism is a config.toml
[model.<name>] block redirected via GROK_HOME. pi/omp's PI_CONFIG_DIR
does nothing for either (grepped pi's entire bundled source — the string
appears nowhere); the real redirect is the child process's own HOME, and
both need `models` as an array of {id} objects, not an object keyed by
id (silently loaded zero models otherwise).

deployment_plan.md, PR.md, docs/custom-model-endpoints.md, and CLAUDE.md
updated with the final confidence table reflecting all of the above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017HqNWfmtBU2KN29SvSVWB3
This commit is contained in:
Devvyn
2026-09-13 17:42:35 +08:00
co-authored by Claude Sonnet 5
parent 41416566aa
commit 61779745aa
14 changed files with 1061 additions and 770 deletions
+42
View File
@@ -0,0 +1,42 @@
/**
* @fileoverview The one IO wrapper around `custom-model-injection.ts`'s pure
* `ConfigDirInjection` output — deliberately split out so that file, the
* discovery routes, and `scripts/test-local-llm-harnesses.ts` (via tsx) can
* all share EXACTLY one "write these files, merge this env" implementation.
* Before this existed, the route and the standalone script each carried
* their own copy of this logic, which is exactly the kind of drift the CLI
* registry's "declare once, consume everywhere" design exists to prevent —
* see deployment_plan.md and the "dynamic to support cli-registry changes"
* requirement it was written against.
*/
import { mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { join, dirname } from 'node:path';
import type { ConfigDirInjection } from './custom-model-injection.js';
/**
* Writes a `ConfigDirInjection`'s files under `baseDir` and returns the full
* envOverrides object a caller should merge into the session/process env
* (the dir-redirect var plus any `extraEnv` the config file references by
* name). Never touches anything outside `baseDir` — the caller is
* responsible for choosing an isolated directory (never the user's real
* `~/.codex`, `~/.pi`, etc.).
*/
export function applyConfigDirInjection(baseDir: string, injection: ConfigDirInjection): Record<string, string> {
for (const file of injection.files) {
const filePath = join(baseDir, file.relPath);
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, file.content, 'utf8');
}
return { [injection.dirEnvVar]: baseDir, ...injection.extraEnv };
}
/** Best-effort recursive removal of a previously-written configDir. Never throws. */
export function removeConfigDir(dir: string | undefined): void {
if (!dir) return;
try {
rmSync(dir, { recursive: true, force: true });
} catch {
// best-effort cleanup only
}
}