[fix] default parameter run records parameters in session for replay[fix] remove default parameters for several cubes
This commit is contained in:
Benjamin Diedrichsen
@@ -4,26 +4,32 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
## What this repo is
|
||||
|
||||
A pnpm workspace holding two independently published CLIs plus the pyinfra
|
||||
deployment units one of them runs:
|
||||
A pnpm workspace holding two independently published CLIs, the authoring package
|
||||
their deployment units are written against, and one bundle of those units:
|
||||
|
||||
| Path | Package | Binary | Role |
|
||||
| ----------------- | ------------------- | -------- | -------------------------------------------------------- |
|
||||
| `packages/nopy` | `@bitsquare/nopy` | `nopy` | interactive pyinfra script management and execution |
|
||||
| `packages/keyman` | `@bitsquare/keyman` | `keyman` | SSH key management, shelling out to `age` / `ssh-keygen` |
|
||||
| `cubes/` | — | — | the deployment units `nopy` runs (not published) |
|
||||
| Path | Package | Binary | Role |
|
||||
| --------------------- | ---------------------- | -------- | -------------------------------------------------------- |
|
||||
| `packages/nopy` | `@bitsquare/nopy` | `nopy` | interactive pyinfra script management and execution |
|
||||
| `packages/keyman` | `@bitsquare/keyman` | `keyman` | SSH key management, shelling out to `age` / `ssh-keygen` |
|
||||
| `packages/nopy-cube` | `@bitsquare/nopy-cube` | — | the authoring surface a `manifest.mjs` imports |
|
||||
| `packages/cubes-core` | `@bitsquare/cubes-core`| — | the core cube bundle (22 cubes), no TypeScript |
|
||||
|
||||
The root package is private; only `packages/*` ship. The two packages do not
|
||||
depend on each other.
|
||||
The root package is private; everything under `packages/` ships. `keyman` stands
|
||||
alone, but `nopy` and `cubes-core` both depend on `nopy-cube` (`workspace:*`), so
|
||||
publish order matters — see *Releasing*.
|
||||
|
||||
`cubes-core` is consumed the way a third party would consume it: the root
|
||||
`.nopyrc.json` names it in `cubePackages`, and the loader reads it out of
|
||||
`node_modules`. There is no `cubes/` directory at the repo root any more.
|
||||
|
||||
## Commands
|
||||
|
||||
```sh
|
||||
pnpm install # also installs the git hooks via simple-git-hooks
|
||||
pnpm run build # tsc --build across both packages (project references)
|
||||
pnpm run typecheck # tsc --build --noEmit
|
||||
pnpm run build # tsc --build across the TS packages (project references)
|
||||
pnpm run typecheck # tsc --build (see below — it really does emit)
|
||||
pnpm run lint # biome check . (lint:fix / lint:ci variants)
|
||||
pnpm test # vitest run, both packages
|
||||
pnpm test # vitest run, every package with tests
|
||||
pnpm run test:coverage # vitest with the coverage gate
|
||||
pnpm run coverage:summary # renders the last coverage run as a Markdown table
|
||||
```
|
||||
@@ -41,6 +47,12 @@ pnpm --filter @bitsquare/keyman run keyman
|
||||
`typescript` is the 7.x native compiler, so `tsc` *is* the fast one — there is no
|
||||
separate `tsgo` binary.
|
||||
|
||||
`typecheck` is plain `tsc --build`, not `--noEmit`. Once a project has
|
||||
`references`, `--noEmit` is rejected outright (TS6310: *referenced project may
|
||||
not disable emit*) — a composite project has to emit the declarations its
|
||||
dependents read. So the typecheck writes `dist` as a side effect; it is
|
||||
gitignored, and the upside is that the gate now also proves the build works.
|
||||
|
||||
## Verification gate
|
||||
|
||||
`lint:ci` → `typecheck` → `test:coverage` is one gate, run in three places: the
|
||||
@@ -55,7 +67,14 @@ locally and on the runner. Barrel files (`src/index.ts`, `src/cubes/index.ts`,
|
||||
`src/nopy.cubes.ts`) and the Commander argv wiring (`src/*.cli.ts`) are excluded;
|
||||
adding logic to those files means moving it somewhere covered.
|
||||
|
||||
Both packages set `pool: 'forks'` because tests use `process.chdir()` — most
|
||||
nopy's vitest config aliases `@bitsquare/nopy-cube` to that package's **source**,
|
||||
not to the workspace link (which points at a `dist` that only exists after a
|
||||
build), so the gate does not depend on build ordering and can never run against a
|
||||
stale artefact. The same config excludes `**/nopy-cube/**` from coverage — without
|
||||
it nopy's numbers absorb another package's files. `cubes-core` has no tests of its
|
||||
own; the loader tests in nopy cover the contract it implements.
|
||||
|
||||
The three TS packages set `pool: 'forks'` because tests use `process.chdir()` — most
|
||||
loader/config tests build a throwaway tree under `os.tmpdir()` and chdir into it,
|
||||
since discovery is driven entirely by the working directory.
|
||||
|
||||
@@ -68,22 +87,43 @@ One pass per invocation, `nopy.main.ts` orchestrating:
|
||||
Per-property strategy comes from the child's `resolution` block (`merge` is
|
||||
the default: arrays concatenate and dedupe, objects deep-merge; `override`
|
||||
replaces). Only properties listed in `PATH_PROPERTIES` (`cubeDirs`) get
|
||||
relative paths resolved against their own config file's directory. **Throws**
|
||||
relative paths resolved against their own config file's directory;
|
||||
`cubePackages` needs the same origin for a different reason, so each entry is
|
||||
normalised into a `CubePackageRef {spec, from}` — `from` is the directory of
|
||||
the config that named it, which is where the package gets resolved from.
|
||||
**Throws**
|
||||
if no config file exists anywhere — which is why `nopy.cli.ts` calls it lazily
|
||||
inside the action, so `--help`/`--version` work outside a project.
|
||||
2. **`cubes/loader.ts`** — `findCubeDirectories()` unions `config.cubeDirs` with
|
||||
every ancestor directory holding a `.npcubes` marker, then scans each
|
||||
recursively (skipping dotted dirs and `node_modules`). A directory is a cube
|
||||
2. **`cubes/packages.ts`** — `resolveCubePackages()` turns each `CubePackageRef`
|
||||
into a package root plus the directories its `nopy.cubes` field declares.
|
||||
Resolution goes through `createRequire(...).resolve.paths()` + `existsSync`,
|
||||
deliberately bypassing the `exports` map: a bundle ships directories and has
|
||||
no entry point to declare. `existsSync` also follows the symlink pnpm plants
|
||||
at `node_modules/<name>`, which a `readdir` scan skips outright (it reports
|
||||
`isSymbolicLink()`, not `isDirectory()`). A missing package, an unreadable
|
||||
manifest, a missing `nopy.cubes`, a directory that does not exist, and an
|
||||
entry pointing outside the package root are all errors, never silent skips.
|
||||
Duplicate refs are deduped here, last-wins, because `mergeValue` only dedupes
|
||||
arrays of primitives and these are objects.
|
||||
3. **`cubes/loader.ts`** — `findCubeRoots()` unions `config.cubeDirs`, the
|
||||
directories from `cubePackages`, and every ancestor directory holding a
|
||||
`.npcubes` marker, then scans each recursively (skipping dotted dirs and
|
||||
`node_modules`). A directory is a cube
|
||||
when it holds both a manifest (`manifest.mjs` or `*.manifest.mjs`) and a
|
||||
deploy script (`deploy.py` or `*.deploy.py`); manifests are loaded by dynamic
|
||||
`import()`. Cube id = `manifest.id` → a `[id]` prefix in `manifest.name` →
|
||||
the directory basename. Ids are flat and need not mirror the path
|
||||
(`cubes/network/tailscale` declares `net:tailscale`). Duplicate ids and bad
|
||||
manifests become entries in `errors`, which aborts the run.
|
||||
3. **`nopy.workflow.ts`** — picks interactive, file-replay, or history-replay and
|
||||
(`cubes/network/tailscale` declares `net:tailscale`), and they are claimed
|
||||
**globally**, not per source: a duplicate is a hard error naming every
|
||||
claimant, with no precedence rule and no shadowing. Each cube carries a
|
||||
`source` — `{type: 'dir', dir}` or `{type: 'package', packageName, dir}` —
|
||||
which is what makes that error legible when the collision is between a local
|
||||
tree and an installed bundle. Duplicate ids and bad manifests become entries
|
||||
in `errors`, which aborts the run.
|
||||
4. **`nopy.workflow.ts`** — picks interactive, file-replay, or history-replay and
|
||||
normalises all three into a `WorkflowResult`. Replays never re-prompt except
|
||||
for passwords (never persisted) and a missing host.
|
||||
4. **`cubes/dependencies.ts` → `BuildContext.resolveCube()`** — the core.
|
||||
5. **`cubes/dependencies.ts` → `BuildContext.resolveCube()`** — the core.
|
||||
Recursive, per (cube, host): assign params and schema defaults → collect
|
||||
variables (prompt, or read them back from the session on replay) → run
|
||||
`before` hooks → resolve `manifest.dependencies(vars)` (dynamic: it receives
|
||||
@@ -92,33 +132,71 @@ One pass per invocation, `nopy.main.ts` orchestrating:
|
||||
`${cubeId}:${host}` set makes emission idempotent. Hooks get a `HookContext`
|
||||
whose `exec(id, vars)` re-enters `resolveCube`, so a hook can pull in a cube
|
||||
that is not a declared dependency.
|
||||
5. **`nopy.executor.ts`** — runs the built `pyinfra <host> -y --data K=V ... --chdir <cubeDir> <script>`
|
||||
6. **`nopy.executor.ts`** — runs the built `pyinfra <host> -y --data K=V ... --chdir <cubeDir> <script>`
|
||||
commands through execa with inherited stdio, sequentially, stopping at the
|
||||
first failure unless `continueOnError`.
|
||||
|
||||
### Variables
|
||||
|
||||
`Variables` (`nopy.common.ts`) keeps three per-cube scopes plus one global bag.
|
||||
`get(id)` merges them lowest-to-highest: `global` (from config `env`) →
|
||||
`defaults` (Zod `.default()`, and recorded session values on replay) → `prompts`
|
||||
(what the user typed) → `params` (values passed by a dependency spec or hook).
|
||||
`Variables` (`nopy.common.ts`) holds one `Variable` per (cube, key). A `Variable`
|
||||
is a list of `Assignment {value, origin}`, and precedence is the `Origin` rank:
|
||||
`default(0) < env(1) < session(2) < prompt(3) < param(4)`. There are no scope
|
||||
bags — config `env` is seeded per cube as a real assignment, so the old
|
||||
`get('global')` (a cube id that was never a cube) is gone, and a replay assigns at
|
||||
`session` instead of being smuggled into the prompts bag.
|
||||
|
||||
`assignments` is the true history, newest first, never reordered. `ordered` is a
|
||||
**stable** sort of it by rank; `value` / `origin` read its head. The stability is
|
||||
load-bearing: it is what makes same-origin ties resolve to the newest while the
|
||||
displaced value stays visible in the trace. The trace is never persisted.
|
||||
|
||||
`get(id)` returns the effective values (→ the pyinfra command line);
|
||||
`persistable(id)` returns the same minus declared secrets (→ session and history).
|
||||
Every schema key is guaranteed present on the pyinfra side; pyinfra parses
|
||||
`--data` values itself, so `"true"` arrives as a bool and numeric strings as ints.
|
||||
|
||||
A manifest's `secrets: string[]` names schema keys holding sensitive values —
|
||||
validated at load (an entry that is not a schema key aborts the run). Secrets are
|
||||
excluded from `persistable()`, re-prompted on replay via `fillSessionGaps`
|
||||
(`requiredKeys() ∪ secrets`), and masked by `maskCommand()` / `maskVariables()`
|
||||
wherever a command is printed. Deliberately a plain array rather than zod
|
||||
metadata: `.meta()` and `.describe()` live in the per-copy `z.globalRegistry`, so
|
||||
a manifest built by a different zod copy would look up empty — fail-open is
|
||||
tolerable for a prompt label and not for a secret marker. See `docs/REFACTORING.md`
|
||||
items 6 and 7.
|
||||
|
||||
### Cube contract
|
||||
|
||||
A cube directory holds `manifest.mjs` + `deploy.py`; anything else in it is
|
||||
ignored by the loader but reachable from the script, which runs with the cube
|
||||
directory as its cwd. Manifests are ESM, import `cubes.Manifest` from
|
||||
`@bitsquare/nopy`, and declare `id`, `name`, a Zod `schema` (each field
|
||||
directory as its cwd. Manifests are ESM, import `Manifest` from
|
||||
`@bitsquare/nopy-cube`, and declare `id`, `name`, a Zod `schema` (each field
|
||||
`.describe()`d — the description is the prompt label — and `.default()`ed), plus
|
||||
optional `dependencies`/`before`/`after`.
|
||||
optional `secrets`/`dependencies`/`before`/`after`.
|
||||
|
||||
**Gotcha:** those manifests resolve `@bitsquare/nopy` through ordinary Node
|
||||
resolution from the manifest's own directory. Nothing in this repo links the
|
||||
package into `cubes/` or `packages/nopy/cubes/`, so loading them fails with
|
||||
`ERR_MODULE_NOT_FOUND` until you link it (`pnpm --filter @bitsquare/nopy run
|
||||
link:local`, then `npm link @bitsquare/nopy` where you run from).
|
||||
Import from **`@bitsquare/nopy-cube`**, not `@bitsquare/nopy`. The authoring
|
||||
surface is types and a factory, with zod as its only peer — no CLI, no prompts,
|
||||
no process spawning — so a bundle can depend on it without dragging the CLI in.
|
||||
`@bitsquare/nopy` re-exports all of it (`cubes.Manifest`, `cubes.uniqid`, …), so
|
||||
the older form still works; every cube in `packages/cubes-core` has been moved to
|
||||
the new one.
|
||||
|
||||
Manifests are resolved by ordinary Node resolution **from the manifest's own
|
||||
directory**, which used to mean a hand-written local cube failed with
|
||||
`ERR_MODULE_NOT_FOUND` unless you linked the package. `cubes/resolve-hook.mjs`
|
||||
retires that: `loadCubes()` registers a `module.register()` resolve hook that
|
||||
tries normal resolution *first* and only on failure falls back to resolving
|
||||
`@bitsquare/nopy-cube`, `@bitsquare/nopy` and `zod` from the running CLI's own
|
||||
`node_modules`. Ordinary-resolution-first is the load-bearing part — a cube that
|
||||
ships its own zod keeps it. The hook is a convenience, never load-bearing:
|
||||
registration is wrapped in a `try`, and a bundle installed properly never reaches
|
||||
it. `dist/cubes/*.mjs` is copied by the build, not compiled — hence
|
||||
`"build": "tsc && cp src/cubes/*.mjs dist/cubes/"`.
|
||||
|
||||
Its tests must spawn a real `node` child process. Written inside the vitest
|
||||
worker they prove nothing: vite resolves the dynamic import itself, so they pass
|
||||
whether or not the hook is installed — verified by commenting the registration
|
||||
out and watching them stay green.
|
||||
|
||||
## keyman architecture
|
||||
|
||||
@@ -134,7 +212,7 @@ the environment beats the config file. Encryption shells out to `age` /
|
||||
|
||||
Tag-driven, one package at a time; see `README.PUBLISH.md`.
|
||||
|
||||
- Push to `main` → `publish-snapshot.yml` publishes both packages to the Gitea
|
||||
- Push to `main` → `publish-snapshot.yml` publishes every package to the Gitea
|
||||
registry as `<version>-main.<run>.g<sha>` under the `main` dist-tag. The
|
||||
version is set on the runner with `npm pkg set` and never committed.
|
||||
- `git tag <dir>-v<version>` (e.g. `nopy-v1.2.0` — the directory under
|
||||
@@ -145,8 +223,42 @@ Tag-driven, one package at a time; see `README.PUBLISH.md`.
|
||||
|
||||
So: bump `packages/<pkg>/package.json`, land it on `main`, then tag that commit.
|
||||
|
||||
Three things the `workspace:*` links added, all of them non-obvious:
|
||||
|
||||
- **`pnpm publish`, never `npm publish`.** `link-workspace-packages` is unset and
|
||||
pnpm 10+ defaults it to `false`, so `workspace:*` is mandatory in the manifests
|
||||
— and npm does not understand it. `npm pack` ships the literal string and the
|
||||
install fails with `EUNSUPPORTEDPROTOCOL`; `pnpm pack`/`pnpm publish` substitute
|
||||
the real version at pack time. Both directions were measured, not assumed.
|
||||
- **`scripts/verify-pack.mjs`** packs every non-private package and fails if any
|
||||
`workspace:` range survived into a tarball. It runs in both publish workflows.
|
||||
Note `pnpm pack` has no `--ignore-scripts` flag, so `prepack` does rebuild —
|
||||
which means the artefact under test is the one publish ships.
|
||||
- **Order.** `packages/*/` sorts `nopy` before `nopy-cube`, which is backwards.
|
||||
`scripts/publish-order.mjs` topologically sorts over the `workspace:` edges;
|
||||
the snapshot workflow stamps *every* version first and only then publishes in
|
||||
that order, because `pnpm publish` reads the linked package's version at pack
|
||||
time. `release.yml` additionally refuses to ship a package whose linked
|
||||
dependency is not yet on npmjs (`scripts/linked-deps.mjs`) — npmjs is the
|
||||
registry you cannot take a mistake back from.
|
||||
|
||||
## Known drift
|
||||
|
||||
`logConfigToFlags()` is exported and tested but nothing feeds its output into the
|
||||
built pyinfra command, so `log.verbosity` / `log.debug` in `.nopyrc.json`
|
||||
currently have no effect. Treat `docs/REFACTORING.md` as a plan, not a record.
|
||||
|
||||
The publish-lane changes above have been verified locally (pack, npm-install of
|
||||
the tarballs into a throwaway tree, run) but have **never run against the Gitea
|
||||
registry**. Burn a throwaway version there before the first real release.
|
||||
|
||||
Nothing checks that a bundle and the CLI reading it are compatible versions;
|
||||
`nopy.engines` was considered and deferred. `docs/CUBE-PACKAGES.md` is where all
|
||||
of this came from and is now a record of what was built, including what differed
|
||||
from the plan.
|
||||
|
||||
`docs/API.md` predates several refactors and still describes `Cube` and
|
||||
`Manifest` as plain interfaces with a `key` field; the code has a `Cube` class
|
||||
keyed on `id`. The `cubePackages` and `CubeSource` sections added for this work
|
||||
are accurate; treat the rest of that file with suspicion. `DOCS-AUDIT.md` tracks
|
||||
the wider drift.
|
||||
|
||||
Reference in New Issue
Block a user