docs(skill): document the per-CLI availability probes, and guard the family

`GET /api/pi/status` shipped undocumented in the agent skill, and only a human
reading the doc noticed. Turns out none of its five siblings were documented
either, so this adds the whole family in one place: spawning with a mode whose
CLI is absent fails with OPERATION_FAILED rather than falling back, which is
exactly what an agent picking a backend it did not choose needs to know. Pi's
extra `.data.version` is called out, since a false `available:false` there means
an unrelated `pi` is in front on PATH.

On whether the endpoint scanner should also check registered-to-documented:
measured, and NO for the general case. The skill documents 34 of 217 registered
endpoints deliberately (it is an agent guide, not an API reference), so a blanket
reverse check needs a 183-entry allowlist that would fail CI on unrelated route
work and get appended to mechanically, which is worse than the gap it closes.
Grouping by path shape does not save it either: the families that yields are
things like `DELETE /api/<any>/:id`, lumping cases, webviews and docker hosts
together, and it would not have caught this gap anyway (the family had zero
documented members).

What IS cheap is a family the schema can enumerate with no allowlist: the new
assertion derives the agent modes from the Zod enum and requires each one's
`/api/<mode>/status` to be documented, so a seventh backend fails here until it
is. The sibling scanner still proves the other direction, that nothing documented
is a 404. Both mutation-checked: dropping pi's probe fails the new guard, and
documenting a nonexistent probe fails the old one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer
2026-08-13 17:55:43 +02:00
parent 86c78fece3
commit 86234db1ef
3 changed files with 34 additions and 2 deletions
+23 -1
View File
@@ -11,7 +11,17 @@
* Two rules, both derived from the RUNTIME source of truth (the Zod enum in schemas.ts,
* not a copy):
*
* 1. The `mode ∈ a|b|c` enumeration in endpoints.md is the mode list, exactly.
* 1. The `mode ∈ a|b|c` enumeration in endpoints.md is the mode list, exactly, and the
* per-CLI availability probe (`GET /api/<mode>/status`) is documented for every
* agent mode. That second half is the narrow, family-scoped answer to "should the
* endpoint scanner also check registered-to-documented?". In general it should not:
* the skill documents 34 of 217 registered endpoints on purpose (it is an agent
* guide, not an API reference), so a blanket reverse check needs a 183-entry
* allowlist that fails CI on unrelated routes and gets appended to mechanically.
* Grouping by path shape does not rescue it either: the families that produces are
* things like `DELETE /api/<any>/:id`, which lumps cases, webviews and docker hosts
* together. A family the SCHEMA can enumerate is the exception, since it needs no
* allowlist at all.
* 2. Any prose enumeration of 3+ distinct modes must be COMPLETE with respect to the
* external CLIs: those lists exist to describe what `isExternalCliMode()` gates
* (no Claude transcript, no hooks, no Claude-format parsers), so naming some but
@@ -72,6 +82,18 @@ describe('agent skill run-mode lists', () => {
expect(EXTERNAL_MODES.length).toBeGreaterThan(1);
});
it('documents the CLI availability probe for every agent mode', () => {
// The gap this closes: /api/pi/status shipped undocumented and only a human reading
// the doc noticed, because the sibling scanner (agent-skill-endpoints-doc.test.ts)
// only checks documented -> registered. Derived from the schema, so a seventh
// backend fails here until its probe is documented; the sibling test still proves
// the reverse, that nothing documented here is a 404.
const doc = readFileSync(join(SKILL_DIR, 'reference/endpoints.md'), 'utf-8');
const documented = new Set([...doc.matchAll(/\bGET\s+\/api(?:\/v1)?\/([a-z-]+)\/status\b/g)].map((m) => m[1]));
const probeable = MODES.filter((m) => m !== 'shell'); // shell has no CLI to probe
expect([...probeable].filter((m) => !documented.has(m))).toEqual([]);
});
it("documents exactly the accepted modes in endpoints.md's `mode ∈ …` enumeration", () => {
const doc = readFileSync(join(SKILL_DIR, 'reference/endpoints.md'), 'utf-8');
const match = doc.match(/`mode` ∈ `([a-z|]+)`/);