From 86234db1ef542ccc699527e110c048bc49e2aef2 Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Thu, 13 Aug 2026 17:55:43 +0200 Subject: [PATCH] docs(skill): document the per-CLI availability probes, and guard the family `GET /api/pi/status` shipped undocumented in the agent skill, and only a human reading the doc noticed. Turns out none of its five siblings were documented either, so this adds the whole family in one place: spawning with a mode whose CLI is absent fails with OPERATION_FAILED rather than falling back, which is exactly what an agent picking a backend it did not choose needs to know. Pi's extra `.data.version` is called out, since a false `available:false` there means an unrelated `pi` is in front on PATH. On whether the endpoint scanner should also check registered-to-documented: measured, and NO for the general case. The skill documents 34 of 217 registered endpoints deliberately (it is an agent guide, not an API reference), so a blanket reverse check needs a 183-entry allowlist that would fail CI on unrelated route work and get appended to mechanically, which is worse than the gap it closes. Grouping by path shape does not save it either: the families that yields are things like `DELETE /api//:id`, lumping cases, webviews and docker hosts together, and it would not have caught this gap anyway (the family had zero documented members). What IS cheap is a family the schema can enumerate with no allowlist: the new assertion derives the agent modes from the Zod enum and requires each one's `/api//status` to be documented, so a seventh backend fails here until it is. The sibling scanner still proves the other direction, that nothing documented is a 404. Both mutation-checked: dropping pi's probe fails the new guard, and documenting a nonexistent probe fails the old one. Co-Authored-By: Claude Opus 5 (1M context) --- .changeset/ba4bc996.md | 2 +- skills/codeman/reference/endpoints.md | 10 ++++++++++ test/agent-skill-mode-lists.test.ts | 24 +++++++++++++++++++++++- 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/.changeset/ba4bc996.md b/.changeset/ba4bc996.md index ef3f1819..223046b1 100644 --- a/.changeset/ba4bc996.md +++ b/.changeset/ba4bc996.md @@ -13,6 +13,6 @@ Add Pi (pi.dev) as a sixth CLI run mode (#206). - **Pi stays out of `isAltScreenStripMode()`.** Its default TUI renders into the main screen with terminal-owned scrollback and is mouse-aware, so it consumes `\x1b[3J` and the mouse DECSETs that the full strip removes, unlike an Ink TUI repainting in place. Note what exclusion does NOT do: pi is tmux-backed, so it still falls through to the narrow `isMuxAltScreenOnlyStripMode()` strip and its alt-screen toggles are dropped either way. Pi's runtime-switchable fullscreen TUI therefore paints into the main buffer, exactly like vim inside a tmux `shell` session. - **Docker**: pi installs in its own `--ignore-scripts` step so that flag cannot affect the other four CLIs, and its credentials are seeded per-file (`auth.json`, `settings.json`, `trust.json`, `models.json`, `models-store.json`) rather than whole-dir, since `~/.pi/agent` also holds sessions, extensions and installed package trees. - **Local echo**: pi lands on the buffer overlay. Verified that codex's per-keystroke starvation does not reproduce — pi's slash picker re-filters on the whole composer content, so a one-shot flush behaves identically to per-keystroke typing. -- **Mode-list parity**: pi is excluded from the Ralph tracker auto-enable on `POST /api/sessions/:id/interactive` (like every other external CLI, whose output the tracker never parses), carries a `REMOTE_CLI_BIN` entry so a remote-SSH pi session reports its CLI version, and gets its own badge in the desktop home rail instead of rendering like Claude. The packaged agent skill's mode enumerations list pi too, pinned by a new guard that derives the mode set from the Zod schema instead of restating it. +- **Mode-list parity**: pi is excluded from the Ralph tracker auto-enable on `POST /api/sessions/:id/interactive` (like every other external CLI, whose output the tracker never parses), carries a `REMOTE_CLI_BIN` entry so a remote-SSH pi session reports its CLI version, and gets its own badge in the desktop home rail instead of rendering like Claude. The packaged agent skill's mode enumerations list pi too, and it now documents the per-CLI availability probes (`GET /api//status`) that agents should check before spawning a worker on a backend the server may not have installed. Both are pinned by a new guard that derives the mode set from the Zod schema instead of restating it. - **`codeman doctor` and the run mode agree about pi.** The registry entry resolved a bare `which pi` while `pi-cli-resolver` demanded semver output, so the Dependencies panel could report an installed Pi CLI that sessions refuse to launch. Both now share one exported regex, and the registry's new `requireVersionMatch` reports a non-semver `pi` as missing rather than installed. Only pi sets it; every other tool keeps its existing behaviour. - Installer detection, docs (`docs/pi-integration.md`), READMEs, and the architecture invariants are updated. Tests: `test/pi-mode.test.ts` and `test/routes/external-cli-bypass-clamp.test.ts`, plus extensions to the run-mode, mobile-overview, render-index-html, system-routes and local-echo suites. diff --git a/skills/codeman/reference/endpoints.md b/skills/codeman/reference/endpoints.md index 092a6403..9fb93d71 100644 --- a/skills/codeman/reference/endpoints.md +++ b/skills/codeman/reference/endpoints.md @@ -338,6 +338,16 @@ ESC=$(printf '\033') `.data.{sessionId, caseName, casePath}`. Creates the case directory (a real directory on the user's disk) if missing, do not retry it in a loop, and remember the name. +⚠️ A `mode` whose CLI is **not installed on the server** fails the spawn with +`OPERATION_FAILED`; it never falls back to claude. Probe first whenever you did not pick +the mode yourself: `GET /api/v1/claude/status`, `GET /api/v1/opencode/status`, +`GET /api/v1/codex/status`, `GET /api/v1/gemini/status`, `GET /api/v1/antigravity/status` +and `GET /api/v1/pi/status` each return `.data.{available, path}` (no session needed). +Pi's also carries `.data.version`, because `pi` is a short generic name that an unrelated +binary on `$PATH` can shadow: the resolver rejects one whose `--version` is not +semver-shaped, so `available:false` there can mean "a different `pi` is in front" rather +than "nothing is installed". `shell` has no CLI to probe. + ⚠️ **Branch on `.success` before reading `.data.sessionId`.** On any failure the field is absent, `jq -r` prints the literal string `null`, and every later call then targets `/api/v1/sessions/null`, burning the full readiness budget and reporting jq noise diff --git a/test/agent-skill-mode-lists.test.ts b/test/agent-skill-mode-lists.test.ts index de3a73e5..5092a506 100644 --- a/test/agent-skill-mode-lists.test.ts +++ b/test/agent-skill-mode-lists.test.ts @@ -11,7 +11,17 @@ * Two rules, both derived from the RUNTIME source of truth (the Zod enum in schemas.ts, * not a copy): * - * 1. The `mode ∈ a|b|c` enumeration in endpoints.md is the mode list, exactly. + * 1. The `mode ∈ a|b|c` enumeration in endpoints.md is the mode list, exactly, and the + * per-CLI availability probe (`GET /api//status`) is documented for every + * agent mode. That second half is the narrow, family-scoped answer to "should the + * endpoint scanner also check registered-to-documented?". In general it should not: + * the skill documents 34 of 217 registered endpoints on purpose (it is an agent + * guide, not an API reference), so a blanket reverse check needs a 183-entry + * allowlist that fails CI on unrelated routes and gets appended to mechanically. + * Grouping by path shape does not rescue it either: the families that produces are + * things like `DELETE /api//:id`, which lumps cases, webviews and docker hosts + * together. A family the SCHEMA can enumerate is the exception, since it needs no + * allowlist at all. * 2. Any prose enumeration of 3+ distinct modes must be COMPLETE with respect to the * external CLIs: those lists exist to describe what `isExternalCliMode()` gates * (no Claude transcript, no hooks, no Claude-format parsers), so naming some but @@ -72,6 +82,18 @@ describe('agent skill run-mode lists', () => { expect(EXTERNAL_MODES.length).toBeGreaterThan(1); }); + it('documents the CLI availability probe for every agent mode', () => { + // The gap this closes: /api/pi/status shipped undocumented and only a human reading + // the doc noticed, because the sibling scanner (agent-skill-endpoints-doc.test.ts) + // only checks documented -> registered. Derived from the schema, so a seventh + // backend fails here until its probe is documented; the sibling test still proves + // the reverse, that nothing documented here is a 404. + const doc = readFileSync(join(SKILL_DIR, 'reference/endpoints.md'), 'utf-8'); + const documented = new Set([...doc.matchAll(/\bGET\s+\/api(?:\/v1)?\/([a-z-]+)\/status\b/g)].map((m) => m[1])); + const probeable = MODES.filter((m) => m !== 'shell'); // shell has no CLI to probe + expect([...probeable].filter((m) => !documented.has(m))).toEqual([]); + }); + it("documents exactly the accepted modes in endpoints.md's `mode ∈ …` enumeration", () => { const doc = readFileSync(join(SKILL_DIR, 'reference/endpoints.md'), 'utf-8'); const match = doc.match(/`mode` ∈ `([a-z|]+)`/);