Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Benjamin Diedrichsen
|
6ecb2c366f | ||
|
Benjamin Diedrichsen
|
ac050c4459 | ||
|
Benjamin Diedrichsen
|
30d93dddc5 | ||
|
Benjamin Diedrichsen
|
5ed68c0065 |
@@ -81,6 +81,13 @@ jobs:
|
||||
echo "::endgroup::"
|
||||
done
|
||||
|
||||
- name: Verify the packed manifests
|
||||
# `workspace:*` is mandatory in the manifests but meaningless to npm, so
|
||||
# a range that survives into a tarball is an install failure for every
|
||||
# consumer. The publish workflows run this too; running it here is what
|
||||
# puts the failure on the pull request instead of on the release.
|
||||
run: node scripts/verify-pack.mjs
|
||||
|
||||
- name: Upload coverage reports
|
||||
if: always()
|
||||
continue-on-error: true
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# Every commit that lands on `main` publishes a prerelease of both packages to
|
||||
# the Gitea npm registry under the `main` dist-tag:
|
||||
# Every commit that lands on `main` publishes a prerelease of every publishable
|
||||
# package to the Gitea npm registry under the `main` dist-tag:
|
||||
#
|
||||
# pnpm add @bitsquare/nopy@main
|
||||
#
|
||||
@@ -78,6 +78,11 @@ jobs:
|
||||
# Explicit, so the publish step can skip lifecycle scripts entirely.
|
||||
run: pnpm run build
|
||||
|
||||
- name: Verify the packed manifests
|
||||
# Packages link to each other with `workspace:*`, which npm cannot
|
||||
# install. Proves on the tarball that pack rewrote it.
|
||||
run: node scripts/verify-pack.mjs
|
||||
|
||||
- name: Authenticate against the Gitea registry
|
||||
run: |
|
||||
set -euo pipefail
|
||||
@@ -97,23 +102,35 @@ jobs:
|
||||
export npm_config_userconfig="$NPMRC"
|
||||
: "${GITHUB_STEP_SUMMARY:=/dev/null}"
|
||||
short_sha=$(git rev-parse --short=7 HEAD)
|
||||
# Dependencies first, so the registry never briefly holds a package
|
||||
# whose dependency has not landed yet.
|
||||
dirs=$(node scripts/publish-order.mjs)
|
||||
|
||||
for dir in packages/*/; do
|
||||
name=$(node -p "require('./${dir}package.json').name")
|
||||
base=$(node -p "require('./${dir}package.json').version")
|
||||
# Pass 1: stamp every manifest before anything is packed. `pnpm
|
||||
# publish` substitutes `workspace:*` with the version the linked
|
||||
# package declares at pack time, so nopy-cube has to be carrying its
|
||||
# snapshot version by the time nopy is packed.
|
||||
for dir in $dirs; do
|
||||
base=$(node -p "require('./${dir}/package.json').version")
|
||||
# `g` prefix keeps the identifier a valid semver one even when the
|
||||
# abbreviated sha happens to be all digits.
|
||||
version="${base}-main.${{ github.run_number }}.g${short_sha}"
|
||||
(cd "$dir" && npm pkg set "version=${version}")
|
||||
done
|
||||
|
||||
# Pass 2: publish.
|
||||
for dir in $dirs; do
|
||||
name=$(node -p "require('./${dir}/package.json').name")
|
||||
version=$(node -p "require('./${dir}/package.json').version")
|
||||
|
||||
echo "::group::${name}@${version}"
|
||||
if npm view "${name}@${version}" version --registry "$REGISTRY" >/dev/null 2>&1; then
|
||||
echo "Already published — skipping (this is a re-run of the same workflow)."
|
||||
else
|
||||
(
|
||||
cd "$dir"
|
||||
npm pkg set "version=${version}"
|
||||
npm publish --ignore-scripts --tag main --registry "$REGISTRY"
|
||||
)
|
||||
# pnpm, not npm: npm ships `workspace:*` verbatim and the install
|
||||
# then fails with EUNSUPPORTEDPROTOCOL. --no-git-checks because
|
||||
# stamping the versions above left the tree dirty.
|
||||
(cd "$dir" && pnpm publish --ignore-scripts --no-git-checks --tag main --registry "$REGISTRY")
|
||||
fi
|
||||
echo "::endgroup::"
|
||||
|
||||
|
||||
@@ -1,10 +1,15 @@
|
||||
# Tag-driven release of a single package.
|
||||
#
|
||||
# git tag nopy-v1.2.0 && git push origin nopy-v1.2.0
|
||||
# git tag nopy-cube-v1.2.0 && git push origin nopy-cube-v1.2.0
|
||||
# git tag keyman-v1.2.0 && git push origin keyman-v1.2.0
|
||||
#
|
||||
# The tag is the source of truth for *which* package ships; package.json is the
|
||||
# source of truth for the version, and the two must agree or the run fails.
|
||||
#
|
||||
# Packages that link to each other release dependency-first — `nopy-cube` before
|
||||
# `nopy` — because the linked version is resolved at pack time. The run refuses
|
||||
# to publish otherwise.
|
||||
# A version with a prerelease part (1.2.0-rc.1) publishes under `next` instead
|
||||
# of `latest`.
|
||||
#
|
||||
@@ -108,6 +113,31 @@ jobs:
|
||||
- name: Install
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Check the linked workspace packages are already released
|
||||
env:
|
||||
NAME: ${{ steps.target.outputs.name }}
|
||||
DIR: ${{ steps.target.outputs.dir }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# `pnpm publish` turns `workspace:*` into the version the linked
|
||||
# package declares at this commit. If that version is not on the
|
||||
# registry yet, the release installs to a broken tree — and npmjs
|
||||
# only lets you unpublish for 72 hours. Release the dependency first:
|
||||
# nopy-cube, then nopy, then any bundle.
|
||||
#
|
||||
# npmjs only: it is the irreversible one, and it needs no credentials
|
||||
# to read, which this step does not have yet.
|
||||
missing=0
|
||||
for spec in $(node scripts/linked-deps.mjs "$DIR" | tr ' ' '@'); do
|
||||
if npm view "$spec" version --registry "$NPMJS_REGISTRY" >/dev/null 2>&1; then
|
||||
echo "${spec} is published"
|
||||
else
|
||||
echo "::error::${NAME} depends on ${spec}, which is not on npmjs. Release it first."
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
exit "$missing"
|
||||
|
||||
- name: Lint
|
||||
run: pnpm run lint:ci
|
||||
|
||||
@@ -121,6 +151,11 @@ jobs:
|
||||
# Explicit, so the publish steps can skip lifecycle scripts entirely.
|
||||
run: pnpm run build
|
||||
|
||||
- name: Verify the packed manifests
|
||||
# Packages link to each other with `workspace:*`, which npm cannot
|
||||
# install. Proves on the tarball that pack rewrote it.
|
||||
run: node scripts/verify-pack.mjs
|
||||
|
||||
- name: Publish to the Gitea registry
|
||||
env:
|
||||
NAME: ${{ steps.target.outputs.name }}
|
||||
@@ -139,7 +174,10 @@ jobs:
|
||||
if npm view "${NAME}@${VERSION}" version --registry "$GITEA_REGISTRY" >/dev/null 2>&1; then
|
||||
echo "${NAME}@${VERSION} is already on Gitea — skipping."
|
||||
else
|
||||
(cd "$DIR" && npm publish --ignore-scripts --tag "$DIST_TAG" --registry "$GITEA_REGISTRY")
|
||||
# pnpm, not npm: npm ships `workspace:*` verbatim and the install
|
||||
# then fails with EUNSUPPORTEDPROTOCOL. --no-git-checks because a
|
||||
# tag build is a detached HEAD.
|
||||
(cd "$DIR" && pnpm publish --ignore-scripts --no-git-checks --tag "$DIST_TAG" --registry "$GITEA_REGISTRY")
|
||||
fi
|
||||
|
||||
- name: Publish to npmjs
|
||||
@@ -162,7 +200,7 @@ jobs:
|
||||
else
|
||||
# No --provenance: that needs GitHub Actions OIDC, which Gitea has no
|
||||
# equivalent for.
|
||||
(cd "$DIR" && npm publish --ignore-scripts --tag "$DIST_TAG" --access public --registry "$NPMJS_REGISTRY")
|
||||
(cd "$DIR" && pnpm publish --ignore-scripts --no-git-checks --tag "$DIST_TAG" --access public --registry "$NPMJS_REGISTRY")
|
||||
fi
|
||||
|
||||
- name: Remove the registry credentials
|
||||
|
||||
+2
-1
@@ -1,6 +1,7 @@
|
||||
{
|
||||
"hosts": [],
|
||||
"cubeDirs": ["./cubes"],
|
||||
"cubeDirs": [],
|
||||
"cubePackages": ["@bitsquare/cubes-core"],
|
||||
"env": {},
|
||||
"log": {
|
||||
"verbosity": "info",
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## What this repo is
|
||||
|
||||
A pnpm workspace holding two independently published CLIs, the authoring package
|
||||
their deployment units are written against, and one bundle of those units:
|
||||
|
||||
| Path | Package | Binary | Role |
|
||||
| --------------------- | ---------------------- | -------- | -------------------------------------------------------- |
|
||||
| `packages/nopy` | `@bitsquare/nopy` | `nopy` | interactive pyinfra script management and execution |
|
||||
| `packages/keyman` | `@bitsquare/keyman` | `keyman` | SSH key management, shelling out to `age` / `ssh-keygen` |
|
||||
| `packages/nopy-cube` | `@bitsquare/nopy-cube` | — | the authoring surface a `manifest.mjs` imports |
|
||||
| `packages/cubes-core` | `@bitsquare/cubes-core`| — | the core cube bundle (22 cubes), no TypeScript |
|
||||
|
||||
The root package is private; everything under `packages/` ships. `keyman` stands
|
||||
alone, but `nopy` and `cubes-core` both depend on `nopy-cube` (`workspace:*`), so
|
||||
publish order matters — see *Releasing*.
|
||||
|
||||
`cubes-core` is consumed the way a third party would consume it: the root
|
||||
`.nopyrc.json` names it in `cubePackages`, and the loader reads it out of
|
||||
`node_modules`. There is no `cubes/` directory at the repo root any more.
|
||||
|
||||
## Commands
|
||||
|
||||
```sh
|
||||
pnpm install # also installs the git hooks via simple-git-hooks
|
||||
pnpm run build # tsc --build across the TS packages (project references)
|
||||
pnpm run typecheck # tsc --build (see below — it really does emit)
|
||||
pnpm run lint # biome check . (lint:fix / lint:ci variants)
|
||||
pnpm test # vitest run, every package with tests
|
||||
pnpm run test:coverage # vitest with the coverage gate
|
||||
pnpm run coverage:summary # renders the last coverage run as a Markdown table
|
||||
```
|
||||
|
||||
Single package / single test:
|
||||
|
||||
```sh
|
||||
pnpm --filter @bitsquare/nopy run test tests/config.test.ts # one file
|
||||
pnpm --filter @bitsquare/nopy run test -t "merges configs" # by test name
|
||||
pnpm --filter @bitsquare/nopy run test:watch
|
||||
pnpm --filter @bitsquare/nopy run nopy # run the CLI from source via tsx
|
||||
pnpm --filter @bitsquare/keyman run keyman
|
||||
```
|
||||
|
||||
`typescript` is the 7.x native compiler, so `tsc` *is* the fast one — there is no
|
||||
separate `tsgo` binary.
|
||||
|
||||
`typecheck` is plain `tsc --build`, not `--noEmit`. Once a project has
|
||||
`references`, `--noEmit` is rejected outright (TS6310: *referenced project may
|
||||
not disable emit*) — a composite project has to emit the declarations its
|
||||
dependents read. So the typecheck writes `dist` as a side effect; it is
|
||||
gitignored, and the upside is that the gate now also proves the build works.
|
||||
|
||||
## Verification gate
|
||||
|
||||
`lint:ci` → `typecheck` → `test:coverage` is one gate, run in three places: the
|
||||
`pre-push` hook, `ci.yml` (non-`main`), and `publish-snapshot.yml` (`main`).
|
||||
`pre-commit` runs Biome with fixes on staged files only. Bypass with
|
||||
`SKIP_SIMPLE_GIT_HOOKS=1`; re-install after editing the hook config with
|
||||
`pnpm exec simple-git-hooks`.
|
||||
|
||||
Coverage thresholds live in each package's `vitest.config.ts` (85 % branches and
|
||||
functions, 80 % lines and statements), not in a CI flag — they fail identically
|
||||
locally and on the runner. Barrel files (`src/index.ts`, `src/cubes/index.ts`,
|
||||
`src/nopy.cubes.ts`) and the Commander argv wiring (`src/*.cli.ts`) are excluded;
|
||||
adding logic to those files means moving it somewhere covered.
|
||||
|
||||
nopy's vitest config aliases `@bitsquare/nopy-cube` to that package's **source**,
|
||||
not to the workspace link (which points at a `dist` that only exists after a
|
||||
build), so the gate does not depend on build ordering and can never run against a
|
||||
stale artefact. The same config excludes `**/nopy-cube/**` from coverage — without
|
||||
it nopy's numbers absorb another package's files. `cubes-core` has no tests of its
|
||||
own; the loader tests in nopy cover the contract it implements.
|
||||
|
||||
The three TS packages set `pool: 'forks'` because tests use `process.chdir()` — most
|
||||
loader/config tests build a throwaway tree under `os.tmpdir()` and chdir into it,
|
||||
since discovery is driven entirely by the working directory.
|
||||
|
||||
## nopy architecture
|
||||
|
||||
One pass per invocation, `nopy.main.ts` orchestrating:
|
||||
|
||||
1. **`nopy.config.ts`** — `loadConfig()` walks up from `process.cwd()` collecting
|
||||
every `.nopyrc.json` plus `~/.nopyrc.json`, then merges them root-first.
|
||||
Per-property strategy comes from the child's `resolution` block (`merge` is
|
||||
the default: arrays concatenate and dedupe, objects deep-merge; `override`
|
||||
replaces). Only properties listed in `PATH_PROPERTIES` (`cubeDirs`) get
|
||||
relative paths resolved against their own config file's directory;
|
||||
`cubePackages` needs the same origin for a different reason, so each entry is
|
||||
normalised into a `CubePackageRef {spec, from}` — `from` is the directory of
|
||||
the config that named it, which is where the package gets resolved from.
|
||||
**Throws**
|
||||
if no config file exists anywhere — which is why `nopy.cli.ts` calls it lazily
|
||||
inside the action, so `--help`/`--version` work outside a project.
|
||||
2. **`cubes/packages.ts`** — `resolveCubePackages()` turns each `CubePackageRef`
|
||||
into a package root plus the directories its `nopy.cubes` field declares.
|
||||
Resolution goes through `createRequire(...).resolve.paths()` + `existsSync`,
|
||||
deliberately bypassing the `exports` map: a bundle ships directories and has
|
||||
no entry point to declare. `existsSync` also follows the symlink pnpm plants
|
||||
at `node_modules/<name>`, which a `readdir` scan skips outright (it reports
|
||||
`isSymbolicLink()`, not `isDirectory()`). A missing package, an unreadable
|
||||
manifest, a missing `nopy.cubes`, a directory that does not exist, and an
|
||||
entry pointing outside the package root are all errors, never silent skips.
|
||||
Duplicate refs are deduped here, last-wins, because `mergeValue` only dedupes
|
||||
arrays of primitives and these are objects.
|
||||
3. **`cubes/loader.ts`** — `findCubeRoots()` unions `config.cubeDirs`, the
|
||||
directories from `cubePackages`, and every ancestor directory holding a
|
||||
`.npcubes` marker, then scans each recursively (skipping dotted dirs and
|
||||
`node_modules`). A directory is a cube
|
||||
when it holds both a manifest (`manifest.mjs` or `*.manifest.mjs`) and a
|
||||
deploy script (`deploy.py` or `*.deploy.py`); manifests are loaded by dynamic
|
||||
`import()`. Cube id = `manifest.id` → a `[id]` prefix in `manifest.name` →
|
||||
the directory basename. Ids are flat and need not mirror the path
|
||||
(`cubes/network/tailscale` declares `net:tailscale`), and they are claimed
|
||||
**globally**, not per source: a duplicate is a hard error naming every
|
||||
claimant, with no precedence rule and no shadowing. Each cube carries a
|
||||
`source` — `{type: 'dir', dir}` or `{type: 'package', packageName, dir}` —
|
||||
which is what makes that error legible when the collision is between a local
|
||||
tree and an installed bundle. Duplicate ids and bad manifests become entries
|
||||
in `errors`, which aborts the run.
|
||||
4. **`nopy.workflow.ts`** — picks interactive, file-replay, or history-replay and
|
||||
normalises all three into a `WorkflowResult`. Replays never re-prompt except
|
||||
for passwords (never persisted) and a missing host.
|
||||
5. **`cubes/dependencies.ts` → `BuildContext.resolveCube()`** — the core.
|
||||
Recursive, per (cube, host): assign params and schema defaults → collect
|
||||
variables (prompt, or read them back from the session on replay) → run
|
||||
`before` hooks → resolve `manifest.dependencies(vars)` (dynamic: it receives
|
||||
the *collected* variables) → emit the deploy call → run `after` hooks. There
|
||||
is no separate topological sort; ordering falls out of the recursion, and a
|
||||
`${cubeId}:${host}` set makes emission idempotent. Hooks get a `HookContext`
|
||||
whose `exec(id, vars)` re-enters `resolveCube`, so a hook can pull in a cube
|
||||
that is not a declared dependency.
|
||||
6. **`nopy.executor.ts`** — runs the built `pyinfra <host> -y --data K=V ... --chdir <cubeDir> <script>`
|
||||
commands through execa with inherited stdio, sequentially, stopping at the
|
||||
first failure unless `continueOnError`.
|
||||
|
||||
### Variables
|
||||
|
||||
`Variables` (`nopy.common.ts`) holds one `Variable` per (cube, key). A `Variable`
|
||||
is a list of `Assignment {value, origin}`, and precedence is the `Origin` rank:
|
||||
`default(0) < env(1) < session(2) < prompt(3) < param(4)`. There are no scope
|
||||
bags — config `env` is seeded per cube as a real assignment, so the old
|
||||
`get('global')` (a cube id that was never a cube) is gone, and a replay assigns at
|
||||
`session` instead of being smuggled into the prompts bag.
|
||||
|
||||
`assignments` is the true history, newest first, never reordered. `ordered` is a
|
||||
**stable** sort of it by rank; `value` / `origin` read its head. The stability is
|
||||
load-bearing: it is what makes same-origin ties resolve to the newest while the
|
||||
displaced value stays visible in the trace. The trace is never persisted.
|
||||
|
||||
`get(id)` returns the effective values (→ the pyinfra command line);
|
||||
`persistable(id)` returns the same minus declared secrets (→ session and history).
|
||||
Every schema key is guaranteed present on the pyinfra side; pyinfra parses
|
||||
`--data` values itself, so `"true"` arrives as a bool and numeric strings as ints.
|
||||
|
||||
A manifest's `secrets: string[]` names schema keys holding sensitive values —
|
||||
validated at load (an entry that is not a schema key aborts the run). Secrets are
|
||||
excluded from `persistable()`, re-prompted on replay via `fillSessionGaps`
|
||||
(`requiredKeys() ∪ secrets`), and masked by `maskCommand()` / `maskVariables()`
|
||||
wherever a command is printed. Deliberately a plain array rather than zod
|
||||
metadata: `.meta()` and `.describe()` live in the per-copy `z.globalRegistry`, so
|
||||
a manifest built by a different zod copy would look up empty — fail-open is
|
||||
tolerable for a prompt label and not for a secret marker. See `docs/REFACTORING.md`
|
||||
items 6 and 7.
|
||||
|
||||
### Cube contract
|
||||
|
||||
A cube directory holds `manifest.mjs` + `deploy.py`; anything else in it is
|
||||
ignored by the loader but reachable from the script, which runs with the cube
|
||||
directory as its cwd. Manifests are ESM, import `Manifest` from
|
||||
`@bitsquare/nopy-cube`, and declare `id`, `name`, a Zod `schema` (each field
|
||||
`.describe()`d — the description is the prompt label — and `.default()`ed), plus
|
||||
optional `secrets`/`dependencies`/`before`/`after`.
|
||||
|
||||
Import from **`@bitsquare/nopy-cube`**, not `@bitsquare/nopy`. The authoring
|
||||
surface is types and a factory, with zod as its only peer — no CLI, no prompts,
|
||||
no process spawning — so a bundle can depend on it without dragging the CLI in.
|
||||
`@bitsquare/nopy` re-exports all of it (`cubes.Manifest`, `cubes.uniqid`, …), so
|
||||
the older form still works; every cube in `packages/cubes-core` has been moved to
|
||||
the new one.
|
||||
|
||||
Manifests are resolved by ordinary Node resolution **from the manifest's own
|
||||
directory**, which used to mean a hand-written local cube failed with
|
||||
`ERR_MODULE_NOT_FOUND` unless you linked the package. `cubes/resolve-hook.mjs`
|
||||
retires that: `loadCubes()` registers a `module.register()` resolve hook that
|
||||
tries normal resolution *first* and only on failure falls back to resolving
|
||||
`@bitsquare/nopy-cube`, `@bitsquare/nopy` and `zod` from the running CLI's own
|
||||
`node_modules`. Ordinary-resolution-first is the load-bearing part — a cube that
|
||||
ships its own zod keeps it. The hook is a convenience, never load-bearing:
|
||||
registration is wrapped in a `try`, and a bundle installed properly never reaches
|
||||
it. `dist/cubes/*.mjs` is copied by the build, not compiled — hence
|
||||
`"build": "tsc && cp src/cubes/*.mjs dist/cubes/"`.
|
||||
|
||||
Its tests must spawn a real `node` child process. Written inside the vitest
|
||||
worker they prove nothing: vite resolves the dynamic import itself, so they pass
|
||||
whether or not the hook is installed — verified by commenting the registration
|
||||
out and watching them stay green.
|
||||
|
||||
## keyman architecture
|
||||
|
||||
Much smaller: `keyman.cli.ts` (argv, plus a `--print-config` escape hatch) →
|
||||
`keyman.main.ts`, an inquirer menu loop dispatching to one module per operation
|
||||
(`list`/`copy`/`generate`/`encrypt`/`decrypt`). `keyman.config.ts` mirrors nopy's
|
||||
upward-traversal + `resolution` merge for `.keymanrc.json`, but validates the
|
||||
result with Zod and falls back to defaults instead of throwing. `VAULT_ROOT` in
|
||||
the environment beats the config file. Encryption shells out to `age` /
|
||||
`age-keygen` / `ssh-keygen`, which must be on `PATH`.
|
||||
|
||||
## Releasing
|
||||
|
||||
Tag-driven, one package at a time; see `README.PUBLISH.md`.
|
||||
|
||||
- Push to `main` → `publish-snapshot.yml` publishes every package to the Gitea
|
||||
registry as `<version>-main.<run>.g<sha>` under the `main` dist-tag. The
|
||||
version is set on the runner with `npm pkg set` and never committed.
|
||||
- `git tag <dir>-v<version>` (e.g. `nopy-v1.2.0` — the directory under
|
||||
`packages/`, not the npm name) → `release.yml` publishes to Gitea *and* npmjs.
|
||||
The tag chooses the package, `package.json` supplies the version, and the run
|
||||
fails if they disagree. A prerelease version goes out as `next`, otherwise
|
||||
`latest`.
|
||||
|
||||
So: bump `packages/<pkg>/package.json`, land it on `main`, then tag that commit.
|
||||
|
||||
Three things the `workspace:*` links added, all of them non-obvious:
|
||||
|
||||
- **`pnpm publish`, never `npm publish`.** `link-workspace-packages` is unset and
|
||||
pnpm 10+ defaults it to `false`, so `workspace:*` is mandatory in the manifests
|
||||
— and npm does not understand it. `npm pack` ships the literal string and the
|
||||
install fails with `EUNSUPPORTEDPROTOCOL`; `pnpm pack`/`pnpm publish` substitute
|
||||
the real version at pack time. Both directions were measured, not assumed.
|
||||
- **`scripts/verify-pack.mjs`** packs every non-private package and fails if any
|
||||
`workspace:` range survived into a tarball. It runs in both publish workflows.
|
||||
Note `pnpm pack` has no `--ignore-scripts` flag, so `prepack` does rebuild —
|
||||
which means the artefact under test is the one publish ships.
|
||||
- **Order.** `packages/*/` sorts `nopy` before `nopy-cube`, which is backwards.
|
||||
`scripts/publish-order.mjs` topologically sorts over the `workspace:` edges;
|
||||
the snapshot workflow stamps *every* version first and only then publishes in
|
||||
that order, because `pnpm publish` reads the linked package's version at pack
|
||||
time. `release.yml` additionally refuses to ship a package whose linked
|
||||
dependency is not yet on npmjs (`scripts/linked-deps.mjs`) — npmjs is the
|
||||
registry you cannot take a mistake back from.
|
||||
|
||||
## Known drift
|
||||
|
||||
`logConfigToFlags()` is exported and tested but nothing feeds its output into the
|
||||
built pyinfra command, so `log.verbosity` / `log.debug` in `.nopyrc.json`
|
||||
currently have no effect. Treat `docs/REFACTORING.md` as a plan, not a record.
|
||||
|
||||
The publish-lane changes above have been verified locally (pack, npm-install of
|
||||
the tarballs into a throwaway tree, run) but have **never run against the Gitea
|
||||
registry**. Burn a throwaway version there before the first real release.
|
||||
|
||||
Nothing checks that a bundle and the CLI reading it are compatible versions;
|
||||
`nopy.engines` was considered and deferred. `docs/CUBE-PACKAGES.md` is where all
|
||||
of this came from and is now a record of what was built, including what differed
|
||||
from the plan.
|
||||
|
||||
`docs/API.md` predates several refactors and still describes `Cube` and
|
||||
`Manifest` as plain interfaces with a `key` field; the code has a `Cube` class
|
||||
keyed on `id`. The `cubePackages` and `CubeSource` sections added for this work
|
||||
are accurate; treat the rest of that file with suspicion. `DOCS-AUDIT.md` tracks
|
||||
the wider drift.
|
||||
+799
@@ -0,0 +1,799 @@
|
||||
# Documentation audit
|
||||
|
||||
Every claim in the repository's Markdown was checked against the source it
|
||||
describes. Findings are grouped by *kind of divergence*, because the fix differs:
|
||||
a phantom feature needs a decision (build it or delete the docs), a wrong claim
|
||||
needs an edit, a gap needs prose.
|
||||
|
||||
Severity is about what it costs a reader:
|
||||
|
||||
- **🔴 broken** — following the documentation produces a wrong result or a crash.
|
||||
- **🟠 misleading** — the documentation states something the code does not do.
|
||||
- **🟡 gap** — the code does something real that no document mentions.
|
||||
|
||||
Verified against the working tree at commit `fcc1817`. Line numbers are from that
|
||||
state.
|
||||
|
||||
Findings closed since are marked **✅ … fixed** and keep their original text as
|
||||
the record of what was wrong. So far: §1.1 (`--use-defaults`), §2.2
|
||||
(`getDefaults()`), §2.1 (precedence — the second half closed differently than
|
||||
proposed), §4.2 (password on stdout — points 1 and 2 of 3), §4.3 (what a session
|
||||
records), part of §3.5 (dead exports), and one bullet of §6.4.
|
||||
|
||||
---
|
||||
|
||||
## Contents
|
||||
|
||||
- [1. Documented features that do not exist](#1-documented-features-that-do-not-exist)
|
||||
- [2. Documented behaviour that differs from the code](#2-documented-behaviour-that-differs-from-the-code)
|
||||
- [3. `docs/API.md` — systematic drift](#3-docsapimd--systematic-drift)
|
||||
- [4. Undocumented behaviour](#4-undocumented-behaviour)
|
||||
- [5. Cube documentation](#5-cube-documentation)
|
||||
- [6. Defects found while verifying](#6-defects-found-while-verifying)
|
||||
- [7. Checked and accurate](#7-checked-and-accurate)
|
||||
- [Suggested order of attack](#suggested-order-of-attack)
|
||||
|
||||
---
|
||||
|
||||
## 1. Documented features that do not exist
|
||||
|
||||
These are the same class of problem as the `--parallel` flag that was removed
|
||||
earlier: documented in detail, absent from the source.
|
||||
|
||||
### 1.1 ✅ `-D, --use-defaults` does nothing — **fixed**
|
||||
|
||||
> **Resolved.** The flag is now implemented; see `docs/REFACTORING.md` item 5.
|
||||
> `BuildContext.resolveCube` skips the prompts, `env` in `.nopyrc.json` outranks
|
||||
> the schema default so a non-interactive run can be configured, and a cube with
|
||||
> a variable nothing can fill aborts the run by name instead of deploying it
|
||||
> blank. The finding below is kept as the record of what was wrong.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Docs say** | `README.md:291` "Install with defaults (no prompts for customization)"; `docs/API.md:39` "Skip variable prompts, use defaults"; `nopy.cli.ts:54` "Run cubes with default values without prompts" |
|
||||
| **Code does** | Nothing. |
|
||||
|
||||
The option is threaded through four layers and then dropped. `nopy.cli.ts:95` →
|
||||
`nopy.main.ts:125` → `nopy.main.ts:174` → `BuildContext.options.useDefaults`
|
||||
(`cubes/dependencies.ts:35`), where it is **never read**. The only branch that
|
||||
skips prompting is `isSessionReplay` (`cubes/dependencies.ts:62`).
|
||||
`runInteractiveWorkflow` destructures only `useAuthKey` and ignores its
|
||||
`useDefaults` too (`nopy.workflow.ts:50`).
|
||||
|
||||
Every documented `-D` invocation — including
|
||||
`nopy install -D --save-session automated-deployment.nopysession.json`
|
||||
(`README.md:226`), which is presented as the way to do an unattended run —
|
||||
prompts for every variable of every cube.
|
||||
|
||||
```
|
||||
$ grep -rn "useDefaults" packages/nopy/src/
|
||||
nopy.workflow.ts:19 useDefaults?: boolean; # declared
|
||||
nopy.main.ts:94 useDefaults?: boolean; # declared
|
||||
nopy.main.ts:125 useDefaults = false, # defaulted
|
||||
nopy.main.ts:158 { useDefaults, useAuthKey },# passed
|
||||
nopy.main.ts:174 useDefaults, # passed
|
||||
nopy.cli.ts:95 useDefaults: options... # passed
|
||||
cubes/dependencies.ts:35 useDefaults?: boolean; # declared — and that is all
|
||||
```
|
||||
|
||||
### 1.2 🔴 `-j, --json` produces no output on success
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Docs say** | `README.md:18` "**JSON output** for CI/CD integration"; `README.md:360-367` "Machine-readable JSON output for scripting and CI/CD integration"; `docs/API.md:573` |
|
||||
| **Code does** | Emits JSON **only** on failure. |
|
||||
|
||||
`jsonOutput` reaches three places in `nopy.main.ts` (140, 150, 219) and none of
|
||||
them prints a result. It suppresses the config banner, prints
|
||||
`{success: false, errors}` when cube *loading* fails, and suppresses progress
|
||||
lines. The success path at `nopy.main.ts:226-236` returns the `NopyResult` object
|
||||
to the caller, and `nopy.cli.ts:107-110` inspects `result.success` without
|
||||
printing it.
|
||||
|
||||
A CI job running `nopy install --json` gets pyinfra's inherited stdio and nothing
|
||||
machine-readable. The exit code is the only usable signal.
|
||||
|
||||
Related: `--dry-run --json` prints the **text** plan, not JSON.
|
||||
`executeDeployCalls` calls `outputExecutionPlan(calls)` without the `asJson`
|
||||
argument (`nopy.executor.ts:172`), even though the function supports it
|
||||
(`nopy.executor.ts:110`).
|
||||
|
||||
### 1.3 🟠 `log.verbosity` and `log.debug` have no effect
|
||||
|
||||
Pre-existing known drift, recorded in `CLAUDE.md`, but the README still presents
|
||||
it as a working feature — two tables, a recommendation paragraph, and a slot in
|
||||
the config example (`README.md:127-130`, `143-163`).
|
||||
|
||||
`logConfigToFlags()` (`nopy.config.ts:352`) is exported and has 8 unit tests, but
|
||||
nothing calls it. `buildDeployCall` (`cubes/dependencies.ts:105-124`) constructs
|
||||
the pyinfra argv without consulting `config.log` at all.
|
||||
|
||||
This is live in the repo's own config: `packages/nopy/.nopyrc.json` sets
|
||||
`"verbosity": "trace", "debug": true` and gets neither.
|
||||
|
||||
### 1.4 🟠 Manifest `env` property
|
||||
|
||||
`README.md:14` lists "**Default values** with optional customization via manifest
|
||||
`env`". `env` was removed from `Manifest` — see `docs/REFACTORING.md` item 3, and
|
||||
the current interface at `cubes/types.ts:43-56`, which has `id`, `name`,
|
||||
`schema`, `dependencies`, `before`, `after` and nothing else.
|
||||
|
||||
### 1.5 🟠 Topological sorting
|
||||
|
||||
`README.md:11` ("Dependency resolution with **topological sorting**"),
|
||||
`README.md:25` ("Topologically sorts cubes based on dependencies") and
|
||||
`README.md:381` ("because cubes are topologically sorted") describe an algorithm
|
||||
that does not exist.
|
||||
|
||||
There is no sort. `BuildContext.resolveCube` recurses depth-first and pushes each
|
||||
cube after its dependencies, with a `${cubeId}:${host}` set for idempotence
|
||||
(`cubes/dependencies.ts:43-100`, `105-143`). The ordering is a side effect of the
|
||||
recursion order.
|
||||
|
||||
This matters beyond vocabulary: a topological sort detects cycles, and this does
|
||||
not. Two cubes that depend on each other recurse until the stack overflows —
|
||||
`resolvedCubes` is only consulted in `buildDeployCall`, which runs *after* the
|
||||
recursive call. `docs/API.md:160` still promises `Error` "if ... circular
|
||||
dependency detected".
|
||||
|
||||
---
|
||||
|
||||
## 2. Documented behaviour that differs from the code
|
||||
|
||||
### 2.1 ✅ Variable precedence is wrong in both directions — **fixed**
|
||||
|
||||
> **Resolved, though the second pair closed the opposite way to the README's
|
||||
> original claim.** `env` now outranks the Zod defaults, as documented — that was
|
||||
> a prerequisite for `--use-defaults` being configurable at all.
|
||||
>
|
||||
> Dependency/hook params still outrank prompts, deliberately. They do not compete
|
||||
> in practice: `VariableAssignment` leaves out any key a dependency supplied, so
|
||||
> the operator is never asked about it and there is no typed value to override.
|
||||
> The README documents the real order rather than the old promise.
|
||||
>
|
||||
> The underlying complaint — that precedence was the field order of an object
|
||||
> literal and so could be neither named nor questioned — is what
|
||||
> `docs/REFACTORING.md` item 6 addresses. `Origin` now ranks
|
||||
> `default < env < session < prompt < param` as data, and every value carries the
|
||||
> origin it came from.
|
||||
|
||||
`README.md:107-113` stated:
|
||||
|
||||
> **Priority order (lowest to highest):**
|
||||
> 1. Zod schema `.default()` values
|
||||
> 2. Global `env` from `.nopyrc.json`
|
||||
> 3. Accumulated variables from dependencies
|
||||
> 4. User prompts / session replay
|
||||
|
||||
`Variables.get()` (`nopy.common.ts:33-39`) does:
|
||||
|
||||
```typescript
|
||||
return {
|
||||
...this.global, // 1. config env (lowest)
|
||||
...this.defaults[id], // 2. Zod defaults
|
||||
...this.prompts[id], // 3. what the user typed
|
||||
...this.params[id], // 4. dependency / hook (highest)
|
||||
};
|
||||
```
|
||||
|
||||
Two pairs are inverted, and both have consequences:
|
||||
|
||||
- **Zod defaults beat `env`, not the other way round.** So `README.md:114` —
|
||||
"allowing users to override them globally via `.nopyrc.json`" — is backwards.
|
||||
Setting `"env": {"UPDATE": false}` cannot override a cube declaring
|
||||
`.default(true)`; the `env` value is only visible for keys the schema does not
|
||||
define. This is why `KEY_DIR` works in `packages/nopy/.nopyrc.json` (no cube
|
||||
declares it) and why anything else would not.
|
||||
- **Dependency/hook parameters beat user prompts.** A value passed as
|
||||
`[['user:add', {USER: 'deploy'}]]` silently overrides what the operator just
|
||||
typed at the form. The docs promise the opposite.
|
||||
|
||||
### 2.2 ✅ "Every key ... is guaranteed to be present on `host.data`" — not when a field lacks `.default()` — **fixed**
|
||||
|
||||
> **Resolved.** `getDefaults()` falls back to a per-field read, so one required
|
||||
> field no longer wipes out the rest; `VariableAssignment` prompts for every
|
||||
> schema key rather than only the defaulted ones; and `--use-defaults` refuses
|
||||
> to deploy a cube whose required key nothing supplied. Verified against all 22
|
||||
> cubes in `cubes/`: 19 build a complete `-D` run, the 3 below abort by name.
|
||||
>
|
||||
> Re-measured against the 25 cubes now in `packages/cubes-core/cubes`: 20 build a
|
||||
> complete `-D` run and 5 abort by name. Two of the additions are deliberate —
|
||||
> `user:add` lost its `PUBKEY` default (it was a specific personal key), and
|
||||
> `ssh:keygen` inherits that failure because it declares `dependencies: () =>
|
||||
> ['user:add']` and passes no parameters. See §6.6.
|
||||
|
||||
`README.md:99` states it outright; `README.md:105` claims defaults ensure "every
|
||||
cube has a predictable starting state".
|
||||
|
||||
`Cube.getDefaults()` (`cubes/types.ts:106-112`) is:
|
||||
|
||||
```typescript
|
||||
try {
|
||||
return this.manifest.schema.parse({});
|
||||
} catch {
|
||||
return {} as z.infer<Schema>;
|
||||
}
|
||||
```
|
||||
|
||||
One field without a `.default()` makes `parse({})` throw, and the `catch`
|
||||
discards the defaults of **every other field in the cube**. `VariableAssignment`
|
||||
then iterates over that empty object and returns before prompting
|
||||
(`nopy.prompts.ts:175-181`), so the user is never asked. The cube deploys with no
|
||||
`--data` flags at all and every `host.data.X` is `None`.
|
||||
|
||||
Three cubes in this repo are in that state today:
|
||||
|
||||
| Cube | Field(s) without `.default()` | Result |
|
||||
|---|---|---|
|
||||
| `net:wifi:connection` | `SSID`, `PASSWORD` | no prompt, no `--data`, all 4 vars lost |
|
||||
| `service:autostart` | `APP` | no prompt, no `--data`, all 3 vars lost |
|
||||
| `user:edit` | `USER` | no prompt, no `--data`, all 4 vars lost |
|
||||
|
||||
The failure is silent — no error, no warning, just a pyinfra run with an empty
|
||||
data set.
|
||||
|
||||
### 2.3 🔴 `.describe()` before `.default()` loses the prompt label
|
||||
|
||||
`CLAUDE.md` and the cube contract state that each schema field is `.describe()`d
|
||||
and "the description is the prompt label". `nopy.prompts.ts:184-185` reads it as:
|
||||
|
||||
```typescript
|
||||
const zodType = schema[key];
|
||||
const description = zodType?.description || key;
|
||||
```
|
||||
|
||||
In zod 4, `.default()` returns a `ZodDefault` **wrapper** that does not inherit
|
||||
`.description` from the type it wraps. Ordering therefore decides whether the
|
||||
label survives:
|
||||
|
||||
```
|
||||
z.boolean().describe('Update package cache').default(false) → description undefined
|
||||
z.boolean().default(false).describe('Update package cache') → description preserved
|
||||
```
|
||||
|
||||
The README's own manifest example (`README.md:67-68`) uses the losing order, so
|
||||
anyone copying it gets bare `UPDATE` / `PACKAGES` keys as prompt labels instead
|
||||
of the sentences they wrote. `docs/API.md:610` happens to use the working order —
|
||||
the two documents disagree, and neither mentions that it matters.
|
||||
|
||||
15 of the 22 cubes in `cubes/` are affected; among them
|
||||
`net:tailscale` (all 4 fields), `runtime:nodevm` (all 4), `user:add` (all 4),
|
||||
`ssh:keygen` (all 4) and `admin:locale` (all 4).
|
||||
|
||||
### 2.4 🟠 Session files claim `version` and `timestamp` fields
|
||||
|
||||
`README.md:179-181` shows a session with `"version": "1.0.0"` and
|
||||
`"timestamp": "2025-10-13T10:30:00Z"`, and `docs/SESSION_FORMAT.md:305-306`
|
||||
declares both **required** in the `NopySession` interface. Every MJS example in
|
||||
that file sets them.
|
||||
|
||||
`NopySession` (`nopy.session.ts:44-55`) has neither. `createSession`
|
||||
(`nopy.session.ts:183-197`) does not add them, `saveSession` writes the object
|
||||
verbatim (`nopy.session.ts:68-79`), and `loadSession`'s validation
|
||||
(`nopy.session.ts:146-155`) checks only `cubes`, `hosts` and `auth`. The
|
||||
repository's own `packages/nopy/example.nopysession.json` omits both — it does
|
||||
not match the format its own documentation prescribes.
|
||||
|
||||
Consequence: a `version` field implies a compatibility check that does not exist.
|
||||
Nothing reads it, so an incompatible old session fails later and more obscurely
|
||||
than a version check would.
|
||||
|
||||
### 2.5 🟠 Session filename convention does not match `listSessions()`
|
||||
|
||||
The READMEs consistently use `*.nopysession.json` (`README.md:223`, `330`, `338`;
|
||||
`docs/DOCKER.md:54`; the shipped `example.nopysession.json`).
|
||||
`docs/SESSION_FORMAT.md` consistently uses `*.session.json` / `*.session.mjs`.
|
||||
|
||||
`listSessions()` (`nopy.session.ts:173`) matches only the second form:
|
||||
|
||||
```typescript
|
||||
.filter((file) => file.endsWith('.session.json') || file.endsWith('.session.mjs'))
|
||||
```
|
||||
|
||||
`"my-deployment.nopysession.json"` does not end in `".session.json"`, so the
|
||||
file naming the README recommends is invisible to the function documented at
|
||||
`docs/API.md:430`. `loadSession` is unaffected (it switches on `.json`/`.mjs`),
|
||||
so this only bites the listing API.
|
||||
|
||||
### 2.6 🟠 `docs/DOCKER.md` container name contradicts the file it points at
|
||||
|
||||
`docs/DOCKER.md:35` and `:45`:
|
||||
|
||||
> We explicitly name it `nopy-test-container` because the
|
||||
> `example.nopysession.json` is configured to target this specific container name.
|
||||
|
||||
`packages/nopy/example.nopysession.json` targets `@docker/nopy-test-ubuntu` —
|
||||
the *image* tag from the build step, not the container name. Following the guide
|
||||
exactly produces a pyinfra run against a container that does not exist.
|
||||
|
||||
(`packages/nopy/.nopyrc.json` does list `@docker/nopy-test-container` in `hosts`,
|
||||
so the guide was probably written against the config rather than the session.)
|
||||
|
||||
### 2.7 🟠 `docs/HOOKS.md` — hook parameters are not validated
|
||||
|
||||
`docs/HOOKS.md:48` describes the hook's second argument as "The final,
|
||||
**validated** variables for the current cube".
|
||||
|
||||
`cubes/dependencies.ts:71` passes `this.variables.get(cubeId)` — a plain merge of
|
||||
the four scopes. `schema.parse()` is called in exactly one place,
|
||||
`Cube.getDefaults()` on an empty object, to extract defaults. Values from
|
||||
prompts, `env`, dependencies and hooks are never validated against the schema at
|
||||
any point in the pipeline. `coerceValue` (`nopy.prompts.ts:145-159`) type-coerces
|
||||
prompt input, which is not the same as validation and does not apply to the other
|
||||
three scopes.
|
||||
|
||||
### 2.8 🟠 `docs/HOOKS.md` — dependencies can pass variables too
|
||||
|
||||
The comparison table at `docs/HOOKS.md:83` says variable passing is
|
||||
"Inherited from env" for dependencies versus "Explicitly passed via `exec()`" for
|
||||
hooks, presenting explicit parameters as a hook-only capability.
|
||||
|
||||
`DependencySpec` is `string | [id, variables?]` (`cubes/types.ts:23`) and
|
||||
`cubes/dependencies.ts:86-88` unpacks the tuple and forwards it into the same
|
||||
`params` scope that `exec()` writes to. The two mechanisms are identical in this
|
||||
respect; per §2.1 both outrank user prompts.
|
||||
|
||||
### 2.9 🟠 nopy README installation section describes the wrong package manager
|
||||
|
||||
`README.md:248-279` says "This package is part of a **yarn** workspace monorepo",
|
||||
then gives `yarn install`, `yarn workspace @bitsquare/nopy build`,
|
||||
`yarn workspace @bitsquare/nopy nopy`, and `yarn nopy`.
|
||||
|
||||
The repo is a **pnpm** workspace: `packageManager: "pnpm@11.17.0"` in the root
|
||||
manifest, `pnpm-workspace.yaml`, a `pnpm-lock.yaml`, and every other document
|
||||
(root `README.md`, `README.PUBLISH.md`, `CLAUDE.md`) uses pnpm. There is no
|
||||
`yarn.lock`.
|
||||
|
||||
The section is also the wrong content for the file. `README.md` is one of three
|
||||
files shipped in the npm tarball (`files: ["dist", "README.md", "LICENSE"]`), so
|
||||
this is what a reader sees on npmjs.com — build-from-monorepo instructions
|
||||
instead of `npm install -g @bitsquare/nopy`, which is what the root README and
|
||||
`README.PUBLISH.md:314` correctly tell people to run.
|
||||
|
||||
### 2.10 🟠 keyman README: two operations missing, one operation invented
|
||||
|
||||
`packages/keyman/README.md:90-96` lists four menu entries: List, Encrypt,
|
||||
Decrypt, Quit. The menu (`keyman.main.ts:54-61`) has six:
|
||||
|
||||
```
|
||||
📋 List keys 📝 Copy public key 🆕 Generate key
|
||||
🔒 Encrypt keys 🔓 Decrypt keys ❌ Quit
|
||||
```
|
||||
|
||||
`Copy public key` and `Generate key` are undocumented — the latter being the only
|
||||
way to create a key inside the tool, which is why the Quick Start
|
||||
(`packages/keyman/README.md:33`) tells the user to shell out to `ssh-keygen`
|
||||
manually.
|
||||
|
||||
Conversely `packages/keyman/README.md:11` advertises "🔄 Support for key
|
||||
rotation". There is no rotation anywhere: `grep -rn "rotat" packages/keyman/src/`
|
||||
returns nothing.
|
||||
|
||||
`packages/keyman/README.md:93` also says encrypt takes keys "from `vault/tmp/`".
|
||||
`encryptKeys` (`keyman.encrypt.ts:12-22`) unions `~/.ssh` and `vault/tmp`, and
|
||||
offers both in the checkbox.
|
||||
|
||||
### 2.11 🟡 Root README understates the coverage gate
|
||||
|
||||
Root `README.md:57-58` describes "a hard **85 % branch** floor". Both
|
||||
`vitest.config.ts` files set four thresholds: branches 85, functions 85, lines
|
||||
80, statements 80. `README.PUBLISH.md:135` and `CLAUDE.md` both state all four —
|
||||
the root README is the odd one out, and it is the file a new contributor reads
|
||||
first.
|
||||
|
||||
### 2.12 🟡 `docs/DOCKER.md` relative link is broken
|
||||
|
||||
`docs/DOCKER.md:8` links `[README.md](./README.md)`, which resolves to
|
||||
`packages/nopy/docs/README.md` — nonexistent. It should be `../README.md`.
|
||||
|
||||
---
|
||||
|
||||
## 3. `docs/API.md` — systematic drift
|
||||
|
||||
`docs/API.md` documents an earlier architecture. It is not a matter of
|
||||
individual stale lines: the two central type definitions, one whole module, and
|
||||
two of the documented functions describe code that no longer exists. Anyone
|
||||
building against this file writes code that will not compile.
|
||||
|
||||
Recommendation: regenerate rather than patch.
|
||||
|
||||
### 3.1 🔴 Functions that do not exist
|
||||
|
||||
| Documented | Reality |
|
||||
|---|---|
|
||||
| `resolveDependencies(cubes, selectedCubeNames)` (`API.md:142-160`) | No such export. Resolution is `BuildContext.resolveCube` and returns nothing — it accumulates into `deployCalls`. |
|
||||
| `buildDeployCalls(cubeNames, hosts, context)` (`API.md:286-313`) | No such export. The entire "Builder Module" section, and its `BuildResult` interface, describes code replaced by `BuildContext` (`docs/REFACTORING.md` item 2). Still listed in the table of contents at `API.md:12`. |
|
||||
|
||||
### 3.2 🔴 `Cube<Schema>` — wrong shape entirely
|
||||
|
||||
`API.md:75-84` documents an interface with `key`, `dependencies: string[]`,
|
||||
`schema`, `defaults()`, `before`, `after`.
|
||||
|
||||
`Cube` (`cubes/types.ts:88-113`) is a **class**: constructor `(manifest, dir,
|
||||
deployScript)`, getters `id` and `name`, method `getDefaults()`. Everything else
|
||||
lives behind `.manifest`. Not one documented member name is correct — `key` is
|
||||
`id`, `defaults()` is `getDefaults()`, and `dependencies`/`schema`/`before`/
|
||||
`after` are on `cube.manifest`, not on `cube`.
|
||||
|
||||
### 3.3 🔴 `Manifest<Schema>` — wrong shape
|
||||
|
||||
`API.md:92-100` documents `key`, `dependencies: string[]`, `defaults: () => ...`.
|
||||
|
||||
`Manifest` (`cubes/types.ts:43-56`) has `id` (not `key`), no `defaults` member at
|
||||
all, and `dependencies` is a **function of the collected variables**:
|
||||
|
||||
```typescript
|
||||
dependencies?: (variables: z.infer<Schema>) => DependencySpec[];
|
||||
```
|
||||
|
||||
That signature change is the headline of `docs/REFACTORING.md` item 2. The
|
||||
`API.md` example at `:171` does use the function form, so the file contradicts
|
||||
itself two paragraphs apart. Both `Cube` and `Manifest` are additionally shown
|
||||
as generic over `z.AnyZodObject`, which zod 4 removed; the codebase defines
|
||||
`AnyObjectSchema` for exactly this reason (`cubes/types.ts:13`).
|
||||
|
||||
### 3.4 🟠 Incorrect signatures and examples
|
||||
|
||||
| Location | Documented | Actual |
|
||||
|---|---|---|
|
||||
| `API.md:486-492` | `saveConfig(data, local?)`, example passes `false` | `saveConfig(data, configPath?: string)` (`nopy.config.ts:323`). Passing `false` writes nothing useful. |
|
||||
| `API.md:481-484` | Search order `./nopyrc.json` then `~/.nopyrc.json` | Filename is `.nopyrc.json` (leading dot). Home is applied **first** (lowest priority, `nopy.config.ts:117-120`), all ancestors are collected and merged root-first, and the function **throws** when none is found (`nopy.config.ts:285-289`). The `resolution` merge strategy is not mentioned. |
|
||||
| `API.md:534-540` | `VariableAssignment(cube, env)` returning vars | `VariableAssignment(cube, variables: Variables)` returns `Promise<void>` and mutates the `Variables` instance (`nopy.prompts.ts:167`). The example's return value is always `undefined`. |
|
||||
| `API.md:321-331` | `runWorkflow(sessionPath, cubes, config, options?)` | Takes a fifth parameter `replaySession?: NopySession` (`nopy.workflow.ts:206-212`) — the entire history-replay path. |
|
||||
| `API.md:337-344` | `WorkflowResult.cubesWithDependencies` | Field is `selectedCubes` (`nopy.workflow.ts:31`). |
|
||||
| `API.md:202-209` | `DeployCall.dependencies: string[]` | `DependencySpec[]` (`nopy.executor.ts:27`) — and always `[]` in practice (`cubes/dependencies.ts:132`). |
|
||||
| `API.md:452-457` | `NopyConfig` with 4 fields | Missing `history` and `execution` (`nopy.config.ts:68-81`). |
|
||||
| `API.md:37-45` | 7 `NopyOptions` parameters | Missing `printOnly`, `replaySession`, `saveToHistory` (`nopy.main.ts:93-104`). |
|
||||
| `API.md:169-175` | `cubes.Manifest` example | Omits `id`, the field that determines the cube's identity. |
|
||||
|
||||
### 3.5 🟡 Exported and undocumented
|
||||
|
||||
Public API in `src/index.ts` with no `API.md` entry: the entire history module
|
||||
(`addToHistory`, `listHistory`, `getLastSession`, `getSessionById`,
|
||||
`clearHistory`, `removeFromHistory`, `loadHistory`, `saveHistory`,
|
||||
`getHistoryPath`, `formatHistoryList`, `HISTORY_FILE`, `DEFAULT_HISTORY_SIZE`,
|
||||
plus `HistoryEntry` / `SessionHistory`), `BuildContext`,
|
||||
`runSessionReplayWorkflow`, `getConfigPaths`, `findCubeDirectories`, `getCube`,
|
||||
~~`filterInternalVariables`, `separateEnvAndCubeVariables`~~ — those last two were
|
||||
dead on arrival (nothing called them once the session recorder stopped splitting
|
||||
`env` out) and have since been deleted rather than documented. `maskCommand`,
|
||||
`maskVariables`, `Variable`, `Variables`, `MASK` and the `Origin` / `Assignment`
|
||||
types are newly exported and also have no `API.md` entry.
|
||||
|
||||
The CLI cheat-sheet (`API.md:554-578`) omits `-R`, `-H`, `-P`, `--no-history`,
|
||||
and the `history` / `clear-history` commands.
|
||||
|
||||
`ManifestFactory` (`cubes/factories.ts:28`) is marked `@deprecated` but is not
|
||||
re-exported from `cubes/index.ts`, so it is unreachable dead code.
|
||||
|
||||
---
|
||||
|
||||
## 4. Undocumented behaviour
|
||||
|
||||
### 4.1 🔴 Cubes in this repo cannot be loaded
|
||||
|
||||
`CLAUDE.md` records the `@bitsquare/nopy` linking gotcha; the package README does
|
||||
not mention it at all, and the gotcha is incomplete.
|
||||
|
||||
Cube manifests are loaded by dynamic `import()` from their own directory
|
||||
(`cubes/loader.ts:75`), so they resolve their imports through ordinary Node
|
||||
resolution from `cubes/…`. Nothing in the tree provides either dependency:
|
||||
|
||||
```
|
||||
node_modules/@bitsquare/ → absent
|
||||
packages/nopy/node_modules/@bitsquare/ → absent
|
||||
cubes/node_modules/ → absent
|
||||
```
|
||||
|
||||
`@bitsquare/nopy` is the documented half. **`zod` is the other half** — 20 of 22
|
||||
manifests `import { z } from 'zod'`, and that fails independently of the nopy
|
||||
link. Verified by loading every manifest with a resolver hook: with only
|
||||
`@bitsquare/nopy` mapped, 20 of 22 fail `ERR_MODULE_NOT_FOUND: zod`.
|
||||
|
||||
Because `loadCubes` turns each failure into an `errors` entry and `nopy.main.ts:147-152`
|
||||
aborts when `errors.length > 0`, a fresh clone cannot run a single cube. Neither
|
||||
README mentions a setup step.
|
||||
|
||||
### 4.2 🟠 The SSH password is printed in plaintext — **mostly fixed**
|
||||
|
||||
> **Points 1 and 2 resolved; point 3 stands.** `maskCommand()`
|
||||
> (`nopy.executor.ts`) rewrites the SSH `--password` and every `--data` value the
|
||||
> manifest declared a secret, and it is applied at all three places the command
|
||||
> string is printed: the debug log, the dry-run plan, and `--print-only`. The
|
||||
> name heuristic described in point 2 is gone — the manifest's `secrets` array
|
||||
> says which keys are sensitive, so `TOKEN`, `PSK` and `AUTH_KEY` are covered
|
||||
> too, and it no longer matters that a key merely *looks* like a password.
|
||||
>
|
||||
> Point 3 is unchanged and now documented instead: the value still reaches
|
||||
> pyinfra on its command line, so it is visible in `ps`. That is inherent to
|
||||
> pyinfra's `--data` interface, not something nopy can mask. The shell-quoting
|
||||
> concern in the same point is also still open. See `docs/REFACTORING.md` item 7.
|
||||
|
||||
Not stated in any document, and it sits directly against the security notes at
|
||||
`README.md:217` and `:325` (which are narrowly about *storage*, and are correct
|
||||
as far as they go).
|
||||
|
||||
`buildDeployCall` embeds the password in the command string
|
||||
(`cubes/dependencies.ts:112-113`):
|
||||
|
||||
```typescript
|
||||
parts.push(`--user ${this.auth.username} --password ${this.auth.password}`);
|
||||
```
|
||||
|
||||
That string is then:
|
||||
|
||||
1. logged at debug level — `log.debug(\`Command: ${commandStr}\`)`
|
||||
(`nopy.executor.ts:76`) — and the `nopy` logger is configured with
|
||||
`lowestLevel: 'debug'` (`nopy.main.ts:41-44`), so it **prints to the console
|
||||
on every deployment**;
|
||||
2. printed unmasked by `--dry-run` — `outputExecutionPlan` masks values whose key
|
||||
contains "password" in the `--data` section (`nopy.executor.ts:134`) but prints
|
||||
`call.command.join(' ')` verbatim one line earlier (`nopy.executor.ts:127`);
|
||||
3. passed through `execa({shell: true})` (`nopy.executor.ts:79`), making it
|
||||
visible in the process list and, without quoting, vulnerable to shell
|
||||
metacharacters in the password.
|
||||
|
||||
### 4.3 ✅ History and session files record only prompted values — **fixed**
|
||||
|
||||
> **Resolved.** `buildDeployCall` records `Variables.persistable(cubeId)` — every
|
||||
> value the cube settled on, whatever its origin — so a `--use-defaults` run no
|
||||
> longer records an empty object, and a replay reproduces the run rather than
|
||||
> re-deriving it from whatever the defaults and `env` say at replay time. The one
|
||||
> deliberate exclusion is a key the manifest declared a secret; those are prompted
|
||||
> for again on replay, and a `-D` replay that would need one fails by name.
|
||||
>
|
||||
> The divergence noted at the end of this finding narrows but does not vanish: a
|
||||
> dependency graph that resolves differently can still produce a different
|
||||
> command, because `param` outranks `session` by design.
|
||||
|
||||
`README.md:414` says an entry records "the variable values that were answered at
|
||||
the prompts" — accurate, but the consequence is not drawn out.
|
||||
`buildDeployCall` records `this.variables.get(cubeId, 'prompts')`
|
||||
(`cubes/dependencies.ts:138`) — the prompts scope alone. Values that came from a
|
||||
dependency spec, a hook's `exec()`, `env`, or a schema default are **not** in the
|
||||
entry.
|
||||
|
||||
Combined with §2.1 (dependency params outrank prompts) this means a replay can
|
||||
legitimately produce a different command than the run it replays, if the
|
||||
dependency graph resolved differently.
|
||||
|
||||
### 4.4 🟡 `-P, --print-only` is undocumented
|
||||
|
||||
Implemented (`nopy.cli.ts:61`, `nopy.main.ts:202-213`), listed in the CLI's own
|
||||
help examples (`nopy.cli.ts:38`), and absent from `README.md` and `docs/API.md`.
|
||||
It prints the built pyinfra commands grouped by cube and exits.
|
||||
|
||||
Worth documenting alongside `--dry-run`, since the difference is not obvious:
|
||||
`--print-only` returns a `NopyResult` with `successful: 0` and skips execution
|
||||
entirely, while `--dry-run` goes through the executor.
|
||||
|
||||
### 4.5 🟡 `--save-session` is ignored during a replay
|
||||
|
||||
`nopy.main.ts:191` guards with `saveSessionPath && !workflow.isReplay`, so
|
||||
`nopy install -R -s out.json` writes nothing and says nothing. The
|
||||
"Recording a Session" section (`README.md:219-227`) does not mention it.
|
||||
|
||||
### 4.6 🟡 `.npcubes` is documented but unused in this repo
|
||||
|
||||
`README.md:43` shows a `cubes/.npcubes` marker in the layout diagram and
|
||||
`README.md:244` documents the mechanism. `find . -name .npcubes` returns nothing
|
||||
— discovery here runs entirely off `cubeDirs` in `.nopyrc.json`. The feature
|
||||
exists in `findCubeDirectories` (`cubes/loader.ts:26-30`); the diagram just shows
|
||||
a file that no reader will find if they go looking.
|
||||
|
||||
### 4.7 🟡 `ssh:keyman` depends on a global `env` value
|
||||
|
||||
`cubes/ssh/keyman/deploy.py` reads `host.data.get('KEY_DIR')`, which no manifest
|
||||
declares — it comes from `env` in `.nopyrc.json`. This works (§2.1: `env` is
|
||||
visible for keys the schema does not define) and it is the only cube relying on
|
||||
the mechanism, but nothing documents the coupling. Anyone running that cube from
|
||||
a project without `KEY_DIR` in their config gets `None`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Cube documentation
|
||||
|
||||
Two cubes have **no README at all**: `cubes/admin/hostname` and `cubes/git/clone`
|
||||
(20 of 22 have one).
|
||||
|
||||
### 5.1 🔴 `cubes/service/autostart/README.md` documents a different cube
|
||||
|
||||
The file is titled **"TypeStack Install Cube"** and describes cloning a git
|
||||
repository, `yarn install`, `yarn build`, `docker compose up -d`, and PM2 process
|
||||
management. The manifest (`service:autostart`, "Manage systemd service
|
||||
autostart") does none of that — it has three fields and calls `systemd.service`.
|
||||
|
||||
| README documents | In the schema? |
|
||||
|---|---|
|
||||
| `USER` | ❌ |
|
||||
| `REPO` | ❌ |
|
||||
| `ENV` | ❌ |
|
||||
| `NODE_PATH` | ❌ |
|
||||
| `APP` | ✅ |
|
||||
| `AUTOSTART` | ✅ |
|
||||
| — | `SERVICE_NAME` (undocumented) |
|
||||
|
||||
Every "Requirements" entry (Git, Yarn, Docker, PM2, NVM, SSH keys) is inapplicable.
|
||||
It reads as a leftover from a cube that was split or renamed.
|
||||
|
||||
### 5.2 🟠 `cubes/network/wifi/access-point/README.md` — four wrong parameters
|
||||
|
||||
| README | Manifest |
|
||||
|---|---|
|
||||
| `NETWORK_DEVICE` (default `wlan0`) | does not exist |
|
||||
| `CHANNEL` | does not exist — **but `deploy.py` reads it** (see §6.3) |
|
||||
| `IP_ADDRESS` (default `192.168.50.1`) | field is `AP_IP`, default `192.168.4.1` |
|
||||
| `CONNECTION_NAME` default `net:wifi:ap` | default is `pi-point` |
|
||||
| SSID / PASSWORD listed as "Required" | both have defaults (`PiPoint` / `1223334444`) |
|
||||
|
||||
Both worked examples set keys that will be ignored.
|
||||
|
||||
### 5.3 🟠 Two cubes claim to have no parameters
|
||||
|
||||
| Cube | README says | Schema has |
|
||||
|---|---|---|
|
||||
| `cubes/runtime/docker` | "This cube currently has no configurable parameters." | `DISTRO` (`ubuntu` \| `debian`) |
|
||||
| `cubes/runtime/nodevm` | "This cube currently has no configurable parameters." | `VERSION`, `USER`, `ALIAS`, `GLOBAL_PACKAGES` |
|
||||
|
||||
`nodevm`'s README also describes installing "the latest LTS via the official
|
||||
NodeSource setup script", while the manifest is named "Install nvm and nodejs
|
||||
with global packages" and takes an explicit `VERSION`.
|
||||
|
||||
(`admin/cockpit` and `armor/fail2ban` make the same claim and are correct — both
|
||||
have empty schemas.)
|
||||
|
||||
### 5.4 🟡 Cube id conventions are inconsistent
|
||||
|
||||
- `cubes/caddy/base` declares `id: 'caddy'` while its sibling declares
|
||||
`caddy:spa`. Every other nested cube uses the `group:name` form.
|
||||
- `net:wifi:connection` sets `name: 'network:wifi:connection - Connect to a WiFi
|
||||
network'` and `user:edit` sets `name: 'user:edit - Modify an existing user
|
||||
account'` — the id is baked into the display name. Since the picker renders
|
||||
`${cube.id} - ${cube.name}` (`nopy.prompts.ts:43`), these show as
|
||||
`net:wifi:connection - network:wifi:connection - Connect to a WiFi network`.
|
||||
`README.md:54` documents `[id]`-in-name as a *fallback* for a missing `id`
|
||||
field, not as a prefix to carry alongside one.
|
||||
|
||||
---
|
||||
|
||||
## 6. Defects found while verifying
|
||||
|
||||
Not documentation issues, but found while checking the docs and worth recording.
|
||||
|
||||
### 6.1 🔴 `cubes/service/autostart/deploy.py` cannot run
|
||||
|
||||
```python
|
||||
from pyinfra.operations import systemd # `server` is never imported
|
||||
from pyinfra import host
|
||||
|
||||
APP = host.data.APP # AUTOSTART and SERVICE_NAME never read
|
||||
|
||||
if AUTOSTART: # NameError
|
||||
...
|
||||
server.shell(...) # NameError, in the else branch
|
||||
```
|
||||
|
||||
`AUTOSTART` and `SERVICE_NAME` are declared in the manifest and never pulled off
|
||||
`host.data`; `server` is used but not imported. The script raises `NameError` on
|
||||
the `if`. Per §2.2 this cube also gets no `--data` at all, so it fails twice over.
|
||||
|
||||
### 6.2 🔴 `-H <id>` and `--no-history` share one destination
|
||||
|
||||
Both options write to `options.history` (`nopy.cli.ts:57` and `:64`). Verified
|
||||
with Commander:
|
||||
|
||||
```
|
||||
argv=[] -> {} # undefined → saves
|
||||
argv=["--no-history"] -> {"history":false} # correct
|
||||
argv=["-H","abc123"] -> {"history":"abc123"} # correct
|
||||
argv=["-H","abc123","--no-history"] -> {"history":false} # id destroyed
|
||||
```
|
||||
|
||||
In the last case the `-H` argument is silently discarded and nopy falls through
|
||||
to a full interactive run instead of replaying. `saveToHistory: options.history
|
||||
!== false` (`nopy.cli.ts:104`) works only because the two meanings happen not to
|
||||
collide in the common cases.
|
||||
|
||||
### 6.3 🟠 `access-point/deploy.py` reads an undeclared variable
|
||||
|
||||
`host.data.get('CHANNEL')` — no manifest declares `CHANNEL`, so it is always
|
||||
`None`. The README documents it as a supported optional parameter (§5.2). One of
|
||||
the three has to give.
|
||||
|
||||
### 6.4 🟡 Debug output left in the shipped code
|
||||
|
||||
- ~~`nopy.common.ts:22` — `console.log('Assigning', artefactId, scope, values)`
|
||||
fires on every variable assignment, printing values to the console. Combined
|
||||
with §4.2 this is a second path by which secrets reach stdout.~~ **Removed**
|
||||
alongside the `--use-defaults` work; it would have made an unattended run
|
||||
unreadable. The two other paths in §4.2 are untouched.
|
||||
- `keyman.encrypt.ts:19-20` — `console.log(tmpKeys); console.log(sshKeys);`
|
||||
before the prompt.
|
||||
|
||||
### 6.5 🟡 No cycle detection
|
||||
|
||||
Covered under §1.5. `docs/API.md:160` documents the error; there is no code that
|
||||
raises it. Mutually dependent cubes recurse until the stack overflows.
|
||||
|
||||
### 6.6 🟠 `ssh:keygen` depends on `user:add` but shares nothing with it
|
||||
|
||||
`ssh:keygen` declares `dependencies: () => ['user:add']` with no parameters, so
|
||||
the two cubes resolve their `USER` independently:
|
||||
|
||||
- `ssh:keygen` defaults `USER` to `vagrant`;
|
||||
- `user:add` defaults `USER` to `` user${uniqid(5)} `` — a fresh random name.
|
||||
|
||||
So the dependency creates an account the dependent then ignores, and generates a
|
||||
key for a `vagrant` user it never created. Passing the value through
|
||||
(`[['user:add', {USER}]]`) is what the dependency spec exists for; `param`
|
||||
outranks `default`, so it would take effect.
|
||||
|
||||
Surfaced by removing `user:add`'s `PUBKEY` default: `ssh:keygen` now fails a `-D`
|
||||
run with `Cube "user:add" cannot run with --use-defaults: PUBKEY has no default
|
||||
value`, naming a cube the operator did not select. The underlying mismatch is
|
||||
older than that change and is not fixed here.
|
||||
|
||||
Related: `user:add`'s `USER` default is generated (`` user${uniqid(5)} ``), the
|
||||
same shape as the `PASSWORD` default that was removed. It is recorded in the
|
||||
session, so replays are stable, but each fresh `-D` run still creates a
|
||||
differently-named account.
|
||||
|
||||
---
|
||||
|
||||
## 7. Checked and accurate
|
||||
|
||||
Recording what was verified and found correct, so a future pass need not redo it.
|
||||
|
||||
- **`README.PUBLISH.md`** — checked against `.gitea/workflows/*.yml` and both
|
||||
manifests. Workflow triggers, the `files` array, dist-tag rules, the snapshot
|
||||
version format, `upload-artifact@v3`, `cache@v4`, the `npm pack --dry-run`
|
||||
step, `retention-days: 7`, and all four coverage thresholds are right. The
|
||||
only file that states the coverage gate completely.
|
||||
- **Root `README.md`** — pnpm/corepack, Node ≥ 22 with `.nvmrc` pinning 24, the
|
||||
script table, and both git hooks match the root `package.json`. Only the
|
||||
coverage line is incomplete (§2.11).
|
||||
- **`CLAUDE.md`** — accurate throughout, including the `logConfigToFlags` drift
|
||||
note and the resolution/merge description. Two additions worth making: `zod` is
|
||||
missing from the cube-linking gotcha (§4.1), and `-D` being a no-op (§1.1)
|
||||
belongs under "Known drift".
|
||||
- **`README.md` history section** (`:392-429`) — the recording rules, the
|
||||
fail-then-`-R` flow, replay-does-not-re-record, the four suppression cases, the
|
||||
`defaults`-layer replay semantics, and the `Cube not found` failure mode all
|
||||
check out against `nopy.history.ts`, `nopy.main.ts:195-200` and
|
||||
`cubes/dependencies.ts:62-69`.
|
||||
- **`README.md` continue-on-error section** (`:369-390`) — fail-fast, no
|
||||
rollback, skipped-cubes-absent-from-results, exit code 1, and CLI-over-config
|
||||
precedence match `nopy.executor.ts:180-193` and `nopy.cli.ts:67-68`.
|
||||
- **`README.md` cube layout and discovery** — the directory-pair rule, recursive
|
||||
scan, dotted/`node_modules` skipping, the prefixed `*.manifest.mjs` fallback,
|
||||
and the three-step id resolution match `cubes/loader.ts` exactly.
|
||||
- **pyinfra `--data` type coercion** (`README.md:101`) — correct.
|
||||
- **keyman config** — priority (`VAULT_ROOT` > file > defaults), the four default
|
||||
values, and the vault layout match `keyman.config.ts` and `keyman.encrypt.ts`.
|
||||
|
||||
---
|
||||
|
||||
## Suggested order of attack
|
||||
|
||||
**1 — ~~Decide on the three phantom features.~~ Two left.** §1.1 (`-D`) is
|
||||
**done** — implemented, tested, and verified against every cube in `cubes/`.
|
||||
That closed §2.2 and half of §2.1 with it, since neither could be left standing
|
||||
under a run that never prompts. §1.2 (`--json`) and §1.3 (`log.*`) are still
|
||||
"documented, wired up, never read": each is a small implementation or a small
|
||||
deletion, but neither can stay documented as working.
|
||||
|
||||
**4 — Decide the `.describe()`/`.default()` ordering (§2.3).** Either read
|
||||
through the `ZodDefault` wrapper in `nopy.prompts.ts`, or fix the ordering in all
|
||||
14 manifests and the README example. The first is one line and cannot regress.
|
||||
|
||||
**5 — Regenerate `docs/API.md` (§3).** Too far gone to patch: two core types,
|
||||
one whole module, and two functions describe code that no longer exists.
|
||||
|
||||
**6 — Cube docs (§5) and the two missing READMEs.** `service/autostart` is the
|
||||
worst — its README belongs to a different cube, and its `deploy.py` does not run
|
||||
at all (§6.1).
|
||||
|
||||
**7 — Secrets on stdout (§4.2, §6.4).** The `console.log` in `Variables.assign`
|
||||
is gone. Still open: mask the password in the executor's debug line and in the
|
||||
dry-run plan, and pass `--user`/`--password` as argv rather than interpolating
|
||||
into a shell string.
|
||||
+138
-23
@@ -21,19 +21,44 @@ shipped. If you only want to cut a release, jump to
|
||||
|
||||
## What ships
|
||||
|
||||
| Directory | Package | Binary |
|
||||
| ----------------- | ------------------ | -------- |
|
||||
| `packages/nopy` | `@bitsquare/nopy` | `nopy` |
|
||||
| `packages/keyman` | `@bitsquare/keyman` | `keyman` |
|
||||
| Directory | Package | Binary | Kind |
|
||||
| --------------------- | ----------------------- | -------- | ------------------------ |
|
||||
| `packages/nopy` | `@bitsquare/nopy` | `nopy` | CLI |
|
||||
| `packages/keyman` | `@bitsquare/keyman` | `keyman` | CLI |
|
||||
| `packages/nopy-cube` | `@bitsquare/nopy-cube` | — | library (cube authoring) |
|
||||
| `packages/cubes-core` | `@bitsquare/cubes-core` | — | cube bundle (no build) |
|
||||
|
||||
Both are ESM, both declare `engines.node >= 22`, and both expose a single
|
||||
All are ESM and declare `engines.node >= 22`. The two CLIs expose a single
|
||||
executable through `bin`, so `npm install -g` puts `nopy` / `keyman` on the
|
||||
`PATH`. `cubes/` is not a package and is never published.
|
||||
`PATH`; the other two are libraries you add to a project.
|
||||
|
||||
The tarball contents are pinned by `files: ["dist", "README.md", "LICENSE"]` —
|
||||
sources and tests are not shipped. `publishConfig.access: "public"` is what makes
|
||||
a scoped package publishable to npmjs without an extra flag; the workflows pass
|
||||
`--access public` anyway.
|
||||
The tarball contents are pinned by `files` — for the three TypeScript packages
|
||||
that is `["dist", "README.md", "LICENSE"]`, so sources and tests are not shipped.
|
||||
`cubes-core` ships `["cubes", "!cubes/**/*.log", "README.md", "LICENSE"]`: the
|
||||
negation matters, because a cube that has been run leaves a `pyinfra-debug.log`
|
||||
next to its `deploy.py`, and `.gitignore` does not filter an npm tarball.
|
||||
`publishConfig.access: "public"` is what makes a scoped package publishable to
|
||||
npmjs without an extra flag; the workflows pass `--access public` anyway.
|
||||
|
||||
### Dependencies between them
|
||||
|
||||
`keyman` stands alone. `nopy` and `cubes-core` both depend on `nopy-cube` through
|
||||
`workspace:*`, which drives three rules the rest of this document keeps coming
|
||||
back to:
|
||||
|
||||
1. **Publish with `pnpm`, not `npm`.** `link-workspace-packages` is unset and
|
||||
pnpm 10+ defaults it to `false`, so `workspace:*` is mandatory in the
|
||||
manifests. npm has no idea what that protocol is: `npm pack` copies the string
|
||||
through verbatim and the install fails with `EUNSUPPORTEDPROTOCOL`. `pnpm
|
||||
pack` and `pnpm publish` substitute the concrete version at pack time. Both
|
||||
workflows use `pnpm publish --ignore-scripts --no-git-checks`.
|
||||
2. **`nopy-cube` publishes before anything that depends on it.**
|
||||
`node scripts/publish-order.mjs` prints the publishable directories in
|
||||
dependency order — note that plain alphabetical `packages/*/` gets this
|
||||
backwards, putting `nopy` first.
|
||||
3. **Every packed manifest is checked.** `node scripts/verify-pack.mjs` packs
|
||||
each non-private package and fails if any `workspace:` range survived into the
|
||||
tarball. It runs in both publish workflows, after the build.
|
||||
|
||||
Versions and changelogs are maintained **by hand**. Nothing in CI commits a
|
||||
version bump, opens a release PR, or pushes a tag. A release happens because you
|
||||
@@ -47,7 +72,7 @@ All three live in [`.gitea/workflows`](.gitea/workflows) and run on the
|
||||
| Workflow | Trigger | Publishes |
|
||||
| ---------------------- | -------------------------------- | ------------------------------------------ |
|
||||
| `ci.yml` | pull requests, non-`main` pushes | nothing |
|
||||
| `publish-snapshot.yml` | pushes to `main` | **both** packages → Gitea, tag `main` |
|
||||
| `publish-snapshot.yml` | pushes to `main` | **every** package → Gitea, tag `main` |
|
||||
| `release.yml` | tags matching `*-v*` | **one** package → Gitea **and** npmjs |
|
||||
|
||||
`ci.yml` explicitly excludes `main` and all tags (`branches-ignore` +
|
||||
@@ -64,13 +89,17 @@ half-finished publish is worse than a slow queue.
|
||||
```
|
||||
checkout → pnpm → node → pnpm store cache → install
|
||||
→ lint:ci → typecheck → test:coverage → coverage summary
|
||||
→ build → npm pack --dry-run (per package) → upload coverage
|
||||
→ build → npm pack --dry-run (per package) → verify-pack
|
||||
→ upload coverage
|
||||
```
|
||||
|
||||
The `npm pack --dry-run --ignore-scripts` step prints the exact file list that
|
||||
would be published. It is there to catch a `files` or `bin` entry pointing at
|
||||
something the build no longer produces — a failure that would otherwise only
|
||||
surface after the version is already on a registry and immutable.
|
||||
surface after the version is already on a registry and immutable. `verify-pack`
|
||||
then packs for real and checks no `workspace:` range survived; both publish
|
||||
workflows run it too, but running it here is what puts the failure on the pull
|
||||
request rather than on the release.
|
||||
|
||||
Coverage HTML/JSON reports are uploaded as a `coverage` artifact with a 7-day
|
||||
retention. Both the artifact upload and the store cache are
|
||||
@@ -82,18 +111,25 @@ slower CI rather than broken CI.
|
||||
```
|
||||
checkout → pnpm → node → cache → install
|
||||
→ lint:ci → typecheck → test:coverage → coverage summary → build
|
||||
→ write .npmrc → publish both packages → delete .npmrc
|
||||
→ verify-pack → write .npmrc → publish every package → delete .npmrc
|
||||
```
|
||||
|
||||
One job, no `needs:` barrier, so install and build happen exactly once and
|
||||
nothing has to be passed between jobs as an artifact.
|
||||
|
||||
The publish step is **two passes** over `node scripts/publish-order.mjs`: the
|
||||
first stamps the snapshot version into every manifest with `npm pkg set`, the
|
||||
second publishes. They cannot be one loop — `pnpm publish` reads a linked
|
||||
package's version out of its manifest at pack time, so stamping and publishing
|
||||
one package at a time would bake the *old* `nopy-cube` version into `nopy`'s
|
||||
tarball.
|
||||
|
||||
### `release.yml`
|
||||
|
||||
```
|
||||
checkout → resolve tag → check secrets
|
||||
→ pnpm → node → cache → install
|
||||
→ lint:ci → typecheck → test:coverage → build
|
||||
→ pnpm → node → cache → install → check linked deps are released
|
||||
→ lint:ci → typecheck → test:coverage → build → verify-pack
|
||||
→ publish to Gitea → publish to npmjs → delete .npmrc
|
||||
→ create the Gitea release → step summary
|
||||
```
|
||||
@@ -102,6 +138,14 @@ Tag resolution and the secret check run **before** anything is installed or
|
||||
built, so a malformed tag or a missing token fails in seconds instead of after
|
||||
the whole gate.
|
||||
|
||||
*Check linked deps are released* asks npmjs whether every `workspace:` dependency
|
||||
of the package being released already exists at the version pnpm is about to
|
||||
bake in (`scripts/linked-deps.mjs` → `npm view`). Tagging `nopy-v1.3.0` while
|
||||
`@bitsquare/nopy-cube@1.1.0` is still unpublished would otherwise ship a tarball
|
||||
nobody can install, and npmjs only lets you unpublish for 72 hours. The check is
|
||||
npmjs-only: it runs before any credentials are written, and npmjs is the registry
|
||||
where the mistake is permanent.
|
||||
|
||||
## The verification gate
|
||||
|
||||
The same three commands guard every path into a registry:
|
||||
@@ -143,7 +187,7 @@ a prerelease over `latest` by accident.
|
||||
|
||||
## Snapshots
|
||||
|
||||
Every commit that lands on `main` publishes both packages to the Gitea registry,
|
||||
Every commit that lands on `main` publishes every package to the Gitea registry,
|
||||
versioned as:
|
||||
|
||||
```
|
||||
@@ -175,7 +219,34 @@ edit is discarded with the workspace and is never committed.
|
||||
```
|
||||
|
||||
The tag name is `<directory>-v<version>` — the directory under `packages/`, not
|
||||
the npm name. `nopy-v1.2.0`, not `@bitsquare/nopy-v1.2.0`.
|
||||
the npm name. `nopy-v1.2.0`, not `@bitsquare/nopy-v1.2.0`. All four prefixes work
|
||||
the same way:
|
||||
|
||||
```sh
|
||||
git tag nopy-v1.2.0
|
||||
git tag keyman-v1.2.0
|
||||
git tag nopy-cube-v1.2.0
|
||||
git tag cubes-core-v1.2.0
|
||||
```
|
||||
|
||||
### Ordering when more than one package changed
|
||||
|
||||
Tags are independent, but the dependency graph is not. If a release touches
|
||||
`nopy-cube` *and* something that depends on it, release them in this order,
|
||||
waiting for each run to go green:
|
||||
|
||||
```
|
||||
nopy-cube → nopy, cubes-core (these two are independent of each other)
|
||||
```
|
||||
|
||||
Release `nopy` first and the run stops at the *check linked deps* step, telling
|
||||
you the `nopy-cube` version it wanted is not on npmjs. That is the guard working;
|
||||
release `nopy-cube`, then re-tag. `node scripts/publish-order.mjs` prints the
|
||||
order if you would rather not reason about it.
|
||||
|
||||
Bumping `nopy-cube` means bumping the packages that depend on it in the same
|
||||
change — the `workspace:*` range resolves to whatever version is in the workspace
|
||||
at pack time, so their next release picks it up whether or not you meant it to.
|
||||
|
||||
The tag decides **which** package ships; `package.json` decides the **version**.
|
||||
The workflow re-reads the manifest and refuses to continue if the two disagree:
|
||||
@@ -281,12 +352,21 @@ file written into the workspace can never be committed by accident.
|
||||
|
||||
## Installing the packages
|
||||
|
||||
From npmjs — public, no configuration:
|
||||
From npmjs — public, no configuration. The CLIs go on the `PATH`:
|
||||
|
||||
```sh
|
||||
npm install -g @bitsquare/nopy @bitsquare/keyman
|
||||
```
|
||||
|
||||
The other two go into a project. A cube bundle is a dev dependency of whatever
|
||||
repo describes your infrastructure; `nopy-cube` is only needed if you are writing
|
||||
cubes of your own:
|
||||
|
||||
```sh
|
||||
pnpm add -D @bitsquare/cubes-core # then name it in .nopyrc.json cubePackages
|
||||
pnpm add -D @bitsquare/nopy-cube zod # authoring your own manifests
|
||||
```
|
||||
|
||||
From the Gitea registry, which holds every snapshot plus a mirror of every
|
||||
release. Per-project, in the repo's `.npmrc`:
|
||||
|
||||
@@ -321,9 +401,17 @@ refuses to overwrite an existing version — without the check, a re-run would f
|
||||
on the first registry and never reach the second.
|
||||
|
||||
**The build is explicit, publishes are `--ignore-scripts`.** `prepack` exists for
|
||||
humans running `npm pack` locally; in CI the build has already run as its own
|
||||
step, and repeating it inside `npm publish` would only cost time and add a way
|
||||
for a lifecycle script to change what ships after the gate looked at it.
|
||||
humans packing locally; in CI the build has already run as its own step, and
|
||||
repeating it inside `pnpm publish` would only cost time and add a way for a
|
||||
lifecycle script to change what ships after the gate looked at it.
|
||||
|
||||
**`verify-pack.mjs` checks the artefact, not the source.** Reading `package.json`
|
||||
in the repo would only tell you what you already know — every one of them says
|
||||
`workspace:*`. The question is what pnpm wrote into the tarball, so the script
|
||||
packs, extracts `package/package.json`, and reads the ranges back out. It cannot
|
||||
pass `--ignore-scripts`, because `pnpm pack` has no such flag (only `pnpm publish`
|
||||
does), so `prepack` rebuilds — which at least means the tarball under test is
|
||||
byte-for-byte the one publish would ship.
|
||||
|
||||
**One job per workflow.** No artifact hand-off, no second install, no risk of
|
||||
publishing a tree that a different job built.
|
||||
@@ -352,7 +440,31 @@ See what would actually be in the tarball:
|
||||
|
||||
```sh
|
||||
pnpm run build
|
||||
cd packages/nopy && npm pack --dry-run --ignore-scripts
|
||||
cd packages/nopy && pnpm pack --dry-run
|
||||
```
|
||||
|
||||
Check that no `workspace:` range leaks into a published manifest — the same
|
||||
check CI runs:
|
||||
|
||||
```sh
|
||||
node scripts/verify-pack.mjs
|
||||
node scripts/publish-order.mjs # the order to release in
|
||||
node scripts/linked-deps.mjs packages/nopy # what must be on the registry first
|
||||
```
|
||||
|
||||
Rehearse an install the way a stranger gets one, without publishing anything.
|
||||
Use **npm**, not pnpm: npm is the one that rejects a leaked `workspace:` range,
|
||||
so a clean install here is the real proof.
|
||||
|
||||
```sh
|
||||
pnpm --filter @bitsquare/nopy-cube pack --pack-destination /tmp/tgz
|
||||
pnpm --filter @bitsquare/nopy pack --pack-destination /tmp/tgz
|
||||
pnpm --filter @bitsquare/cubes-core pack --pack-destination /tmp/tgz
|
||||
|
||||
mkdir /tmp/try && cd /tmp/try && npm init -y
|
||||
npm install /tmp/tgz/*.tgz
|
||||
echo '{"hosts":["h"],"cubePackages":["@bitsquare/cubes-core"]}' > .nopyrc.json
|
||||
./node_modules/.bin/nopy install -l session.json -P -D
|
||||
```
|
||||
|
||||
Try the binary as an end user would get it, without publishing:
|
||||
@@ -384,6 +496,9 @@ npm view @bitsquare/nopy@1.2.0 version \
|
||||
| `npm pack --dry-run` step fails | A `files` or `bin` path no longer exists after the build. Fix before it reaches a registry. |
|
||||
| Snapshot workflow green, nothing installable | Snapshots are only on Gitea and only under `@main`. Point the scope at the Gitea registry. |
|
||||
| The release workflow did not trigger | The tag must match `*-v*` and must be pushed (`git push origin <tag>`), not just created. |
|
||||
| `EUNSUPPORTEDPROTOCOL` / `Unsupported URL Type "workspace:"` on install | A `workspace:` range reached a tarball — something published with `npm publish` instead of `pnpm publish`. `verify-pack.mjs` exists to catch this before it ships. |
|
||||
| `... is not published yet on npmjs` before the gate runs | Releasing a package before its `nopy-cube` dependency. Tag and release `nopy-cube` first, then re-tag. |
|
||||
| `verify-pack.mjs` fails locally with a build error | `pnpm pack` runs `prepack`, so a broken build fails the check. Fix the build; there is no skip flag. |
|
||||
|
||||
## Recovering from a bad publish
|
||||
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
id: 'user:add',
|
||||
name: 'Add a user with fish shell and tools',
|
||||
dependencies: () => ['apt:essentials'],
|
||||
schema: z.object({
|
||||
USER: z
|
||||
.string()
|
||||
.describe('Username for the new user account')
|
||||
.default(() => `user${cubes.uniqid(5)}`),
|
||||
PASSWORD: z.string().describe('Password for the new user account').default(cubes.uniqid),
|
||||
GROUPS: z
|
||||
.string()
|
||||
.describe('Comma-separated list of additional groups (e.g., "docker,sudo")')
|
||||
.default(''),
|
||||
PUBKEY: z
|
||||
.string()
|
||||
.describe('SSH public key to authorize for the user')
|
||||
.default(
|
||||
'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICpnZ6IxwQKL1rBE4dy7w5Sd3s2tLFZUDfjH87C1QIlc bdiedrichsen@Benjamins-MBP.lan'
|
||||
),
|
||||
}),
|
||||
});
|
||||
+2
-1
@@ -13,7 +13,7 @@
|
||||
"test": "pnpm -r run test",
|
||||
"test:coverage": "pnpm -r run test:coverage",
|
||||
"coverage:summary": "node scripts/coverage-summary.mjs",
|
||||
"typecheck": "tsc --build --noEmit",
|
||||
"typecheck": "tsc --build",
|
||||
"lint": "biome check .",
|
||||
"lint:fix": "biome check --write .",
|
||||
"lint:ci": "biome ci .",
|
||||
@@ -26,6 +26,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@biomejs/biome": "^2.5.5",
|
||||
"@bitsquare/cubes-core": "workspace:*",
|
||||
"@logtape/logtape": "^2.2.4",
|
||||
"@types/node": "^26.1.1",
|
||||
"simple-git-hooks": "^2.13.1",
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 bitsquare
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,59 @@
|
||||
# @bitsquare/cubes-core
|
||||
|
||||
The core cube bundle for [nopy](https://www.npmjs.com/package/@bitsquare/nopy):
|
||||
base packages, users, SSH, firewalling, networking, web serving and runtimes.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
pnpm add -D @bitsquare/cubes-core
|
||||
```
|
||||
|
||||
Then name it in `.nopyrc.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"hosts": ["web-1"],
|
||||
"cubePackages": ["@bitsquare/cubes-core"]
|
||||
}
|
||||
```
|
||||
|
||||
`nopy` resolves the package from the directory of the config file that named it,
|
||||
reads `nopy.cubes` out of its `package.json`, and scans those directories exactly
|
||||
as it scans a `cubeDirs` entry. Nothing has to be linked or copied.
|
||||
|
||||
## What is in it
|
||||
|
||||
| Area | Cube ids |
|
||||
| ---------- | ------------------------------------------------------------------- |
|
||||
| admin | `admin:cockpit`, `admin:hostname`, `admin:locale` |
|
||||
| packages | `apt:essentials`, `apt:install` |
|
||||
| hardening | `armor:fail2ban`, `armor:ssh`, `armor:ufw` |
|
||||
| web | `caddy`, `caddy:spa` |
|
||||
| source | `git:clone` |
|
||||
| networking | `net:tailscale`, `net:wifi:access-point`, `net:wifi:connection` |
|
||||
| runtimes | `runtime:docker`, `runtime:nodevm` |
|
||||
| services | `service:autostart` |
|
||||
| ssh | `ssh:authorize`, `ssh:keygen`, `ssh:keyman` |
|
||||
| users | `user:add`, `user:edit` |
|
||||
|
||||
Run `nopy` and pick from the list, or `nopy -P` to print the pyinfra commands
|
||||
without executing them. Each cube directory has its own `README.md`.
|
||||
|
||||
## Cube ids are global
|
||||
|
||||
An id such as `apt:essentials` is claimed repo-wide, not per bundle: two cubes
|
||||
with the same id — whichever sources they came from — abort the run with an
|
||||
error naming both. Prefix your own cubes distinctly if you also point
|
||||
`cubeDirs` at a local tree.
|
||||
|
||||
## The bundle is read-only
|
||||
|
||||
Under pnpm the installed files are hardlinked into the global store, so a cube
|
||||
that writes next to its own `deploy.py` corrupts that store for every project on
|
||||
the machine. Cubes here write to `/tmp` or to the remote host, never to their
|
||||
own directory.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'admin:cockpit',
|
||||
name: 'Install cockpit and utils',
|
||||
dependencies: () => [],
|
||||
+2
-2
@@ -1,11 +1,11 @@
|
||||
import { cubes, uniqid } from '@bitsquare/nopy';
|
||||
import { Manifest, uniqid } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
/**
|
||||
* Manifest for the admin:hostname cube.
|
||||
* This cube allows for setting and persistently changing the system's hostname.
|
||||
*/
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'admin:hostname',
|
||||
name: 'Permanently change the hostname',
|
||||
dependencies: () => [],
|
||||
@@ -12,9 +12,9 @@ Configures system keyboard layout permanently by updating `/etc/default/keyboard
|
||||
## Usage
|
||||
|
||||
```javascript
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
name: 'My Host Setup',
|
||||
dependencies: () => [
|
||||
['admin:locale', { LAYOUT: 'de' }]
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'admin:locale',
|
||||
name: 'Configure system locale and keyboard layout',
|
||||
dependencies: () => [],
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'apt:essentials',
|
||||
name: 'Install essential packages',
|
||||
dependencies: () => [],
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'apt:install',
|
||||
name: 'Install packages with apt',
|
||||
dependencies: () => [],
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'armor:fail2ban',
|
||||
name: 'Install and enable fail2ban',
|
||||
dependencies: () => [],
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'armor:ssh',
|
||||
name: 'Secure SSH server by disabling password authentication',
|
||||
dependencies: () => [],
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'armor:ufw',
|
||||
name: 'Activate ufw (uncomplicated firewall)',
|
||||
dependencies: () => ['apt:essentials'],
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'caddy',
|
||||
name: 'Install Caddy webserver',
|
||||
dependencies: () => [],
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'caddy:spa',
|
||||
name: 'Install single page application',
|
||||
dependencies: () => [],
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'git:clone',
|
||||
name: 'Clone a repository',
|
||||
dependencies: () => [],
|
||||
+7
-1
@@ -13,7 +13,7 @@ Installs and authenticates the Tailscale client on a Linux host.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `AUTH_KEY` | `""` | Tailscale Auth Key (recommended to use a 'reusable' or 'ephemeral' key). |
|
||||
| `AUTH_KEY` | `""` | **Secret.** Tailscale Auth Key (recommended to use a 'reusable' or 'ephemeral' key). |
|
||||
| `LOGIN_SERVER` | `https://controlplane.tailscale.com` | The coordination server URL. Set this to your Headscale instance URL if applicable. |
|
||||
| `EXTRA_ARGS` | `""` | Additional flags to pass to `tailscale up` (e.g., `--advertise-exit-node`). |
|
||||
| `FORCE_REAUTH` | `false` | If true, forces the client to re-authenticate. |
|
||||
@@ -25,3 +25,9 @@ nopy install tailscale
|
||||
```
|
||||
|
||||
When prompted, provide your `AUTH_KEY`. If you are using Headscale, also provide the `LOGIN_SERVER` URL.
|
||||
|
||||
`AUTH_KEY` is declared in the manifest's `secrets`, so nopy keeps it out of session
|
||||
and history files and masks it in any command it prints. It is asked for again on
|
||||
replay, and a `--use-defaults` replay refuses rather than joining the tailnet with
|
||||
an empty key. Prefer an ephemeral key regardless — the value is still on pyinfra's
|
||||
command line while the deployment runs.
|
||||
+3
-2
@@ -1,10 +1,11 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'net:tailscale',
|
||||
name: 'Install and authenticate Tailscale',
|
||||
dependencies: () => ['apt:essentials'],
|
||||
secrets: ['AUTH_KEY'],
|
||||
schema: z.object({
|
||||
AUTH_KEY: z.string().describe('Tailscale Auth Key for headless authentication').default(''),
|
||||
LOGIN_SERVER: z
|
||||
+12
-3
@@ -19,10 +19,19 @@ Configures a Linux device as a WiFi Access Point using NetworkManager's `nmcli`
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
### Required
|
||||
> **This section is out of date** — it lists parameters the manifest does not
|
||||
> declare (`NETWORK_DEVICE`, `CHANNEL`, `IP_ADDRESS`) and omits `AP_IP`. Read
|
||||
> `manifest.mjs` for the real list. Tracked as §5 in the repository's
|
||||
> `DOCS-AUDIT.md`.
|
||||
|
||||
- **SSID**: WiFi network name (1-32 characters)
|
||||
- **PASSWORD**: WPA2 password (8-63 characters)
|
||||
### Prompted first
|
||||
|
||||
- **SSID**: WiFi network name (1-32 characters). Defaults to `PiPoint`.
|
||||
- **PASSWORD**: WPA2 password (8-63 characters). Defaults to `1223334444` — a
|
||||
placeholder that should not survive contact with a real network.
|
||||
Declared in the manifest's `secrets`, so nopy keeps it out of session and
|
||||
history files and masks it in printed commands, and re-prompts on replay. The
|
||||
value is still on pyinfra's command line, so it is visible in `ps` during the run.
|
||||
|
||||
### Optional (with defaults)
|
||||
|
||||
+3
-2
@@ -1,10 +1,11 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'net:wifi:access-point',
|
||||
name: 'Configure WiFi Access Point (NetworkManager)',
|
||||
dependencies: () => [],
|
||||
secrets: ['PASSWORD'],
|
||||
schema: z.object({
|
||||
SSID: z.string().min(1).max(32).default('PiPoint').describe('WiFi network name (SSID)'),
|
||||
PASSWORD: z
|
||||
+5
@@ -42,6 +42,11 @@ nopy install network:wifi:connection --env SSID="OfficeWiFi" --env PASSWORD="pas
|
||||
|
||||
## Security Notes
|
||||
|
||||
- `PASSWORD` is declared in the manifest's `secrets`: nopy keeps it out of session
|
||||
and history files and masks it in every command it prints. It is prompted for
|
||||
again on replay.
|
||||
- That covers what nopy writes, not everything. The value is still on pyinfra's
|
||||
command line, so it is visible in `ps` while the deployment runs.
|
||||
- WiFi passwords will be stored in `/etc/NetworkManager/system-connections/` on the target host.
|
||||
- Passing passwords via `--env` may leave them in your local shell history.
|
||||
|
||||
+3
-2
@@ -1,4 +1,4 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
// [agnt://cogen/cogen/network-wifi-connection-1]{cartridge: "ansiblings/cubes", action: "generated", status: "generated"}
|
||||
@@ -7,9 +7,10 @@ import { z } from 'zod';
|
||||
* Manifest for the network:wifi:connection cube.
|
||||
* Configures a WiFi client connection using NetworkManager (nmcli).
|
||||
*/
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'net:wifi:connection',
|
||||
name: 'network:wifi:connection - Connect to a WiFi network',
|
||||
secrets: ['PASSWORD'],
|
||||
schema: z.object({
|
||||
SSID: z.string().min(1).describe('The SSID of the WiFi network to connect to'),
|
||||
PASSWORD: z.string().min(8).describe('The password for the WiFi network'),
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'runtime:docker',
|
||||
name: 'Install docker and tools',
|
||||
dependencies: () => [],
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'runtime:nodevm',
|
||||
name: 'Install nvm and nodejs with global packages',
|
||||
dependencies: () => [],
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'service:autostart',
|
||||
name: 'Manage systemd service autostart',
|
||||
dependencies: () => [],
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'ssh:authorize',
|
||||
name: 'Authorize SSH public key for a user',
|
||||
dependencies: () => [],
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'ssh:keygen',
|
||||
name: 'Generate SSH key for a given $USER',
|
||||
dependencies: () => ['user:add'],
|
||||
@@ -1,7 +1,7 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'ssh:keyman',
|
||||
name: 'Deploy an ssh key managed by keyman',
|
||||
dependencies: () => [],
|
||||
@@ -36,9 +36,15 @@ This cube creates a new user account with a modern shell environment (Fish), SSH
|
||||
- Username for the new user account
|
||||
- Default: `userXXXXX` (randomly generated 5-character suffix)
|
||||
|
||||
- **PASSWORD** (string, auto-generated)
|
||||
- **PASSWORD** (string, **secret**)
|
||||
- Password for the new user account
|
||||
- Default: randomly generated secure password
|
||||
- Default: the literal `changeme` — a placeholder, not a credential. Change it
|
||||
on first login, or pass a real one.
|
||||
- Declared in the manifest's `secrets`, so it is never written to a session or
|
||||
history file and is masked in printed commands. A replay asks for it again.
|
||||
- It used to default to a randomly generated password. That was removed: since
|
||||
the value is not recorded, an unattended run created an account with a
|
||||
credential nobody had seen, and replaying that run produced a different one.
|
||||
|
||||
- **GROUPS** (string, default: `''`)
|
||||
- Comma-separated list of additional groups (e.g., `"docker,sudo"`)
|
||||
@@ -47,9 +53,16 @@ This cube creates a new user account with a modern shell environment (Fish), SSH
|
||||
- `sudo` - Administrative privileges
|
||||
- `www-data` - Web server file access
|
||||
|
||||
- **PUBKEY** (string, has default)
|
||||
- **PUBKEY** (string, **required** — no default)
|
||||
- SSH public key to authorize for the user
|
||||
- Should be your public key for passwordless SSH access
|
||||
- There is deliberately no default. It used to be a specific personal key, so
|
||||
accepting the default authorized *someone else's* key on the new account.
|
||||
No key would be a sensible guess, so the cube asks instead.
|
||||
- Because it is required, `--use-defaults` refuses to run this cube unless
|
||||
`PUBKEY` comes from `env` in `.nopyrc.json`, a dependency, or a hook.
|
||||
- Submitting an empty value at the prompt authorizes no key at all (the account
|
||||
is still created, with password login only).
|
||||
|
||||
## Dependencies
|
||||
|
||||
@@ -7,7 +7,10 @@ USER = host.data.USER
|
||||
HOME_DIR = f"/home/{USER}"
|
||||
TMP_DIR = f"{HOME_DIR}/tmp"
|
||||
PASSWORD = host.data.PASSWORD
|
||||
# An empty submission at the prompt must not become an empty authorized_keys
|
||||
# line, so an absent key means no key rather than a blank one.
|
||||
PUBKEY = host.data.PUBKEY
|
||||
PUBKEYS = [PUBKEY] if PUBKEY and str(PUBKEY).strip() else []
|
||||
GROUPS = list(filter(None, map(str.strip, str(host.data.GROUPS).split())))
|
||||
FISH_PATH = "/usr/bin/fish"
|
||||
FISH_CONFIG_DIR = f"{HOME_DIR}/.config/fish"
|
||||
@@ -31,7 +34,7 @@ server.user(
|
||||
create_home=True,
|
||||
groups=GROUPS,
|
||||
shell=FISH_PATH,
|
||||
public_keys=[PUBKEY],
|
||||
public_keys=PUBKEYS,
|
||||
_sudo=True
|
||||
)
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
import { Manifest, uniqid } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default Manifest({
|
||||
id: 'user:add',
|
||||
name: 'Add a user with fish shell and tools',
|
||||
dependencies: () => ['apt:essentials'],
|
||||
secrets: ['PASSWORD'],
|
||||
schema: z.object({
|
||||
USER: z
|
||||
.string()
|
||||
.describe('Username for the new user account')
|
||||
.default(() => `user${uniqid(5)}`),
|
||||
// A fixed placeholder, not a generated one: the password is never recorded
|
||||
// in a session, so a generated default meant every run produced credentials
|
||||
// nobody had seen and a replay produced different ones again.
|
||||
PASSWORD: z.string().describe('Password for the new user account').default('changeme'),
|
||||
GROUPS: z
|
||||
.string()
|
||||
.describe('Comma-separated list of additional groups (e.g., "docker,sudo")')
|
||||
.default(''),
|
||||
// No default on purpose. This used to carry a specific personal key, which
|
||||
// meant an unattended run authorised someone else's key on the new account.
|
||||
// Leaving it required makes `--use-defaults` refuse by name instead of
|
||||
// guessing, and there is no key that would be a sensible guess.
|
||||
PUBKEY: z.string().describe('SSH public key to authorize for the user'),
|
||||
}),
|
||||
});
|
||||
@@ -18,7 +18,7 @@ This cube allows you to update existing user accounts on the target system. It c
|
||||
| Variable | Type | Description | Required |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `USER` | `string` | The username of the account to modify | Yes |
|
||||
| `PASSWORD` | `string` | New password for the user | No |
|
||||
| `PASSWORD` | `string` | New password for the user. **Secret**: never recorded in a session or history file, masked in printed commands, re-prompted on replay. | No |
|
||||
| `GROUPS` | `string` | Comma-separated list of groups to ADD (e.g., `docker,sudo`) | No |
|
||||
| `GROUPS_ABSENT` | `string` | Comma-separated list of groups to REMOVE | No |
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
// [agnt://cogen/cogen/user-edit-1]{cartridge: "ansiblings/cubes", action: "generated", status: "generated"}
|
||||
@@ -7,10 +7,11 @@ import { z } from 'zod';
|
||||
* Manifest for the user:edit cube.
|
||||
* Allows modifying existing user accounts (password, groups).
|
||||
*/
|
||||
export default cubes.Manifest({
|
||||
export default Manifest({
|
||||
id: 'user:edit',
|
||||
name: 'user:edit - Modify an existing user account',
|
||||
dependencies: () => [],
|
||||
secrets: ['PASSWORD'],
|
||||
schema: z.object({
|
||||
USER: z.string().describe('The username of the account to modify'),
|
||||
PASSWORD: z.string().optional().describe('New password for the user (optional)'),
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"name": "@bitsquare/cubes-core",
|
||||
"version": "1.0.0-alpha0",
|
||||
"description": "The core nopy cube bundle: apt, users, ssh, networking, services and runtimes.",
|
||||
"keywords": [
|
||||
"nopy",
|
||||
"nopy-cubes",
|
||||
"pyinfra",
|
||||
"deployment",
|
||||
"infrastructure"
|
||||
],
|
||||
"license": "MIT",
|
||||
"author": "bitsquare",
|
||||
"type": "module",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://gitea.bitsquare.dev/BitSquare/ansiblings.git",
|
||||
"directory": "packages/cubes-core"
|
||||
},
|
||||
"homepage": "https://gitea.bitsquare.dev/BitSquare/ansiblings/src/branch/main/packages/cubes-core",
|
||||
"bugs": {
|
||||
"url": "https://gitea.bitsquare.dev/BitSquare/ansiblings/issues"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
},
|
||||
"nopy": {
|
||||
"cubes": [
|
||||
"./cubes"
|
||||
]
|
||||
},
|
||||
"files": [
|
||||
"cubes",
|
||||
"!cubes/**/*.log",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"dependencies": {
|
||||
"@bitsquare/nopy-cube": "workspace:*",
|
||||
"zod": "^4.4.3"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 bitsquare
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,69 @@
|
||||
# @bitsquare/nopy-cube
|
||||
|
||||
The authoring surface for [nopy](https://www.npmjs.com/package/@bitsquare/nopy)
|
||||
cubes — the `Manifest` factory, the `Cube` class, and the types around them.
|
||||
|
||||
A cube manifest ships nothing but data, so it should not have to depend on a CLI
|
||||
to describe itself. This package is what a **cube bundle** depends on: no
|
||||
`commander`, no `inquirer`, no `execa`, no process spawning. `@bitsquare/nopy`
|
||||
re-exports everything here, so a manifest that already imports from
|
||||
`@bitsquare/nopy` keeps working unchanged.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
pnpm add @bitsquare/nopy-cube zod
|
||||
```
|
||||
|
||||
`zod` is a **peer dependency** on purpose: the manifest, the schema it builds and
|
||||
the `Manifest` factory should all see the same copy.
|
||||
|
||||
## Writing a manifest
|
||||
|
||||
```js
|
||||
// cubes/net/tailscale/manifest.mjs
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default Manifest({
|
||||
id: 'net:tailscale',
|
||||
name: 'Tailscale',
|
||||
schema: z.object({
|
||||
AUTH_KEY: z.string().describe('Tailscale auth key'),
|
||||
ACCEPT_ROUTES: z.boolean().describe('Accept advertised routes').default(true),
|
||||
}),
|
||||
secrets: ['AUTH_KEY'],
|
||||
dependencies: (vars) => (vars.ACCEPT_ROUTES ? ['net:ip-forwarding'] : []),
|
||||
before: [async (ctx, vars) => ctx.exec('apt:essentials', {})],
|
||||
});
|
||||
```
|
||||
|
||||
Every schema field should carry a `.describe()` — nopy uses it as the prompt
|
||||
label — and a `.default()` wherever a sensible one exists, so `--use-defaults`
|
||||
can run the cube without prompting.
|
||||
|
||||
`secrets` names the schema keys that hold sensitive values. Nopy keeps those out
|
||||
of session and history files and masks them in every command it prints; it does
|
||||
not infer them, so a key nothing declares is recorded and printed in the clear.
|
||||
Each entry must be a key of `schema` — naming anything else is a manifest error.
|
||||
Give a secret a placeholder `.default()` rather than a real credential: a default
|
||||
lives in the manifest, where none of that protection reaches it.
|
||||
|
||||
The manifest lives next to a `deploy.py` in the same directory; together they
|
||||
make a cube. See the
|
||||
[nopy README](https://www.npmjs.com/package/@bitsquare/nopy) for the full cube
|
||||
contract and for how to publish a directory of cubes as a bundle.
|
||||
|
||||
## Exports
|
||||
|
||||
| Export | What it is |
|
||||
| ----------------------------------- | -------------------------------------------------------------- |
|
||||
| `Manifest(opts)` | Builds a manifest, filling in `id`, `schema`, `secrets`, `before`, `after` |
|
||||
| `createManifest` / `manifest` | Aliases of `Manifest` |
|
||||
| `Cube` | A loaded manifest plus its directory; `getDefaults()`, `requiredKeys()`, `secrets`, `isSecret()` |
|
||||
| `zodKind` / `zodInner` | Instance-agnostic zod introspection, safe across zod copies |
|
||||
| `AnyObjectSchema`, `CubeVariables`, `DependencySpec`, `Hook`, `HookContext`, `CubeSource`, `LoadResult` | types |
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"name": "@bitsquare/nopy-cube",
|
||||
"version": "1.0.0-alpha0",
|
||||
"description": "Authoring types for nopy cubes: the Manifest factory and the Cube contract.",
|
||||
"keywords": [
|
||||
"nopy",
|
||||
"pyinfra",
|
||||
"deployment",
|
||||
"infrastructure"
|
||||
],
|
||||
"license": "MIT",
|
||||
"author": "bitsquare",
|
||||
"type": "module",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://gitea.bitsquare.dev/BitSquare/ansiblings.git",
|
||||
"directory": "packages/nopy-cube"
|
||||
},
|
||||
"homepage": "https://gitea.bitsquare.dev/BitSquare/ansiblings/src/branch/main/packages/nopy-cube",
|
||||
"bugs": {
|
||||
"url": "https://gitea.bitsquare.dev/BitSquare/ansiblings/issues"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
},
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./dist/index.d.ts",
|
||||
"default": "./dist/index.js"
|
||||
},
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"dist",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"scripts": {
|
||||
"clean": "rm -rf dist .tsbuildinfo",
|
||||
"build": "tsc",
|
||||
"prepack": "pnpm run build",
|
||||
"link:local": "pnpm run build && npm link",
|
||||
"test": "vitest run",
|
||||
"test:coverage": "vitest run --coverage",
|
||||
"test:watch": "vitest"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"zod": "^4.4.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^26.1.1",
|
||||
"@vitest/coverage-v8": "^4.1.10",
|
||||
"typescript": "^7.0.2",
|
||||
"vitest": "^4.1.10",
|
||||
"zod": "^4.4.3"
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
/**
|
||||
* Factory functions for creating cube configurations
|
||||
* @module cubes/factories
|
||||
* @module factories
|
||||
*/
|
||||
|
||||
import { type AnyObjectSchema, Manifest } from './types.js';
|
||||
@@ -0,0 +1,31 @@
|
||||
/**
|
||||
* @bitsquare/nopy-cube — the authoring surface for nopy cubes.
|
||||
*
|
||||
* Everything a `manifest.mjs` needs and nothing else: no CLI, no prompts, no
|
||||
* process spawning. `@bitsquare/nopy` re-exports all of it, so a manifest can
|
||||
* import from either package.
|
||||
*
|
||||
* @packageDocumentation
|
||||
*/
|
||||
|
||||
export {
|
||||
createManifest,
|
||||
ManifestFactory,
|
||||
manifest,
|
||||
} from './factories.js';
|
||||
export type {
|
||||
AnyObjectSchema,
|
||||
CubeSource,
|
||||
CubeVariables,
|
||||
DependencySpec,
|
||||
Hook,
|
||||
HookContext,
|
||||
LoadResult,
|
||||
} from './types.js';
|
||||
export {
|
||||
Cube,
|
||||
Manifest,
|
||||
zodInner,
|
||||
zodKind,
|
||||
} from './types.js';
|
||||
export { uniqid } from './utils.js';
|
||||
@@ -0,0 +1,224 @@
|
||||
/**
|
||||
* Type definitions for Nopy cubes
|
||||
* @module types
|
||||
*/
|
||||
|
||||
import { z } from 'zod';
|
||||
|
||||
/**
|
||||
* Any object schema, whatever its shape.
|
||||
*
|
||||
* Stands in for zod 3's `z.AnyZodObject`, which zod 4 removed.
|
||||
*/
|
||||
export type AnyObjectSchema = z.ZodObject<Record<string, z.ZodType<any>>>;
|
||||
|
||||
/**
|
||||
* Variables that can be passed to a cube
|
||||
*/
|
||||
export type CubeVariables = Record<string, string | number | boolean>;
|
||||
|
||||
/**
|
||||
* A dependency specification
|
||||
*/
|
||||
export type DependencySpec = string | [id: string, variables?: CubeVariables];
|
||||
|
||||
/**
|
||||
* Context passed to cube hooks for executing other cubes
|
||||
*/
|
||||
export interface HookContext {
|
||||
exec: (key: string, variables: CubeVariables) => Promise<void> | void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Hook function type for before/after cube execution
|
||||
*/
|
||||
export type Hook<Schema extends AnyObjectSchema = AnyObjectSchema> = (
|
||||
ctx: HookContext,
|
||||
variables: z.infer<Schema>
|
||||
) => void | Promise<void>;
|
||||
|
||||
/**
|
||||
* User-defined specification for a cube
|
||||
*/
|
||||
export interface Manifest<Schema extends AnyObjectSchema = AnyObjectSchema> {
|
||||
/** Unique identifier for the cube (used for dependency references) */
|
||||
id: string;
|
||||
/** Human-readable name of the cube */
|
||||
name: string;
|
||||
/** Zod schema for validating cube variables */
|
||||
schema: Schema;
|
||||
/**
|
||||
* Schema keys holding secrets. Their values are never written to a session
|
||||
* file, and are masked wherever a command or a variable would be printed.
|
||||
*
|
||||
* A plain array rather than schema-level metadata on purpose: `.meta()` and
|
||||
* `.describe()` both store into zod's global registry, which is per-copy — a
|
||||
* manifest that builds its schema with its own zod writes the marker into a
|
||||
* registry this process cannot read. A missed `.describe()` costs an ugly
|
||||
* prompt label; a missed secret marker writes a password to disk, so this one
|
||||
* cannot be allowed to fail open. See {@link zodKind} for the same hazard.
|
||||
*/
|
||||
secrets?: string[];
|
||||
/** Dynamic dependency resolver based on collected variables */
|
||||
dependencies?: (variables: z.infer<Schema>) => DependencySpec[];
|
||||
/** Hooks to run before cube execution */
|
||||
before?: Hook<Schema>[];
|
||||
/** Hooks to run after cube execution */
|
||||
after?: Hook<Schema>[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function and namespace for Manifest
|
||||
*/
|
||||
export function Manifest<Schema extends AnyObjectSchema>(
|
||||
opts: Pick<Manifest<Schema>, 'name'> & Partial<Omit<Manifest<Schema>, 'name'>>
|
||||
): Manifest<Schema> {
|
||||
return {
|
||||
id: opts.id ?? '',
|
||||
name: opts.name,
|
||||
schema: opts.schema ?? (z.object({}) as unknown as Schema),
|
||||
secrets: opts.secrets ?? [],
|
||||
dependencies: opts.dependencies,
|
||||
before: opts.before ?? [],
|
||||
after: opts.after ?? [],
|
||||
};
|
||||
}
|
||||
|
||||
export namespace Manifest {
|
||||
/**
|
||||
* Internal create helper
|
||||
*/
|
||||
export function create<Schema extends AnyObjectSchema>(
|
||||
opts: Pick<Manifest<Schema>, 'name'> & Partial<Omit<Manifest<Schema>, 'name'>>
|
||||
): Manifest<Schema> {
|
||||
return Manifest(opts);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* zod's runtime discriminant for a schema node, as a plain string.
|
||||
*
|
||||
* `instanceof z.ZodDefault` compares against the *running* copy of zod. A cube
|
||||
* manifest is free to build its schema with a different copy — its own
|
||||
* dependency, or one shipped inside a bundle — and then every `instanceof`
|
||||
* quietly returns false and the caller falls through to a wrong answer instead
|
||||
* of failing. `def.type` holds across instances, so nothing here may go back to
|
||||
* `instanceof`.
|
||||
*/
|
||||
export function zodKind(zodType: unknown): string {
|
||||
return (zodType as { def: { type: string } }).def.type;
|
||||
}
|
||||
|
||||
/**
|
||||
* The type a wrapper wraps — `.default()`, `.optional()`, `.nullable()`.
|
||||
* Only call this for a node whose {@link zodKind} is one of those.
|
||||
*/
|
||||
export function zodInner(zodType: unknown): z.ZodType {
|
||||
return (zodType as { def: { innerType: z.ZodType } }).def.innerType;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the `.default()` off a schema field, unwrapping the wrappers that may
|
||||
* sit above it (`.default().optional()`, `.default().nullable()`).
|
||||
*
|
||||
* Returns `undefined` for a field that declares no default — which is also how
|
||||
* `requiredKeys()` recognises a field the user has to supply.
|
||||
*/
|
||||
function defaultValueOf(zodType: z.ZodType): unknown {
|
||||
const kind = zodKind(zodType);
|
||||
if (kind === 'default') {
|
||||
// zod 4 exposes `defaultValue` as a getter that already invokes a lazily
|
||||
// declared default; the function branch is insurance against that changing.
|
||||
const { defaultValue } = (zodType as unknown as { def: { defaultValue: unknown } }).def;
|
||||
return typeof defaultValue === 'function' ? defaultValue() : defaultValue;
|
||||
}
|
||||
if (kind === 'optional' || kind === 'nullable') {
|
||||
return defaultValueOf(zodInner(zodType));
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Where a cube was discovered.
|
||||
*
|
||||
* Worth carrying because a cube's own directory does not say how it got into
|
||||
* the run: `/…/node_modules/@acme/cubes-net/cubes/x` could equally have come
|
||||
* from a `cubeDirs` entry pointing straight at it.
|
||||
*/
|
||||
export type CubeSource =
|
||||
/** Found under a `cubeDirs` entry or a `.npcubes` marker, at `dir`. */
|
||||
| { type: 'dir'; dir: string }
|
||||
/** Contributed by a package named in `cubePackages`. */
|
||||
| { type: 'package'; packageName: string; dir: string };
|
||||
|
||||
/**
|
||||
* A fully loaded cube with its filesystem location and runtime state
|
||||
*/
|
||||
export class Cube<Schema extends AnyObjectSchema = AnyObjectSchema> {
|
||||
constructor(
|
||||
public readonly manifest: Manifest<Schema>,
|
||||
public readonly dir: string,
|
||||
public readonly deployScript: string,
|
||||
/** Defaults to the cube's own directory, for cubes built by hand. */
|
||||
public readonly source: CubeSource = { type: 'dir', dir }
|
||||
) {}
|
||||
|
||||
get id(): string {
|
||||
return this.manifest.id;
|
||||
}
|
||||
|
||||
get name(): string {
|
||||
return this.manifest.name;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns default values for the cube's schema.
|
||||
*
|
||||
* Parsing an empty object resolves every default in one go, but it fails
|
||||
* outright as soon as one field has no `.default()`. Falling back to a
|
||||
* per-field read keeps the defaults that *are* declared instead of dropping
|
||||
* the whole set — a single required field used to leave the cube with no
|
||||
* variables at all.
|
||||
*/
|
||||
getDefaults(): z.infer<Schema> {
|
||||
const parsed = this.manifest.schema.safeParse({});
|
||||
if (parsed.success) return parsed.data as z.infer<Schema>;
|
||||
|
||||
const defaults: Record<string, unknown> = {};
|
||||
for (const [key, zodType] of Object.entries(this.manifest.schema.shape)) {
|
||||
const value = defaultValueOf(zodType);
|
||||
if (value !== undefined) defaults[key] = value;
|
||||
}
|
||||
return defaults as z.infer<Schema>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Schema keys that have to be supplied from somewhere: no `.default()`, and
|
||||
* not optional. Nothing else can fill them in, so a run that cannot prompt
|
||||
* has to fail rather than deploy a cube with the value missing.
|
||||
*/
|
||||
requiredKeys(): string[] {
|
||||
return Object.entries(this.manifest.schema.shape)
|
||||
.filter(([, zodType]) => !zodType.safeParse(undefined).success)
|
||||
.map(([key]) => key);
|
||||
}
|
||||
|
||||
/** Schema keys the manifest declared as secrets. */
|
||||
get secrets(): string[] {
|
||||
return this.manifest.secrets ?? [];
|
||||
}
|
||||
|
||||
isSecret(key: string): boolean {
|
||||
return this.secrets.includes(key);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of loading cubes from the filesystem
|
||||
*/
|
||||
export interface LoadResult {
|
||||
/** Map of cube key to Cube object */
|
||||
cubes: Record<string, Cube>;
|
||||
/** List of errors encountered during loading */
|
||||
errors: string[];
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
/**
|
||||
* Utility functions for cubes
|
||||
* @module cubes/utils
|
||||
* @module utils
|
||||
*/
|
||||
|
||||
/**
|
||||
+2
-2
@@ -1,10 +1,10 @@
|
||||
/**
|
||||
* Tests for cubes/factories module
|
||||
* Tests for the manifest factories
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { z } from 'zod';
|
||||
import { createManifest, manifest } from '../src/cubes/factories.js';
|
||||
import { createManifest, manifest } from '../src/factories.js';
|
||||
|
||||
describe('createManifest', () => {
|
||||
it('creates manifest with basic properties', () => {
|
||||
@@ -0,0 +1,29 @@
|
||||
/**
|
||||
* A schema that behaves like zod's but does not share zod's prototypes.
|
||||
*
|
||||
* Once cubes arrive from `node_modules`, the schema a manifest builds may come
|
||||
* from a *second* copy of zod — its own dependency, or one shipped inside a
|
||||
* bundle. Such a schema is structurally identical and `instanceof` blind to it.
|
||||
* Rebuilding the nodes as plain objects reproduces that from inside a single
|
||||
* process, so anything that reads zod's internals stays pinned to `def.type`.
|
||||
*/
|
||||
|
||||
import type { z } from 'zod';
|
||||
|
||||
/** Strips the prototype off a schema node and everything it wraps. */
|
||||
function strip(node: unknown): unknown {
|
||||
const def = { ...(node as { def: Record<string, unknown> }).def };
|
||||
if (def.innerType) def.innerType = strip(def.innerType);
|
||||
return { def };
|
||||
}
|
||||
|
||||
export function foreignZodSchema<S extends z.ZodObject<any>>(schema: S): S {
|
||||
return {
|
||||
// Parsing is not what is under test — delegate it and keep the real
|
||||
// behaviour, so only the introspection path sees the foreign nodes.
|
||||
safeParse: (value: unknown) => schema.safeParse(value),
|
||||
shape: Object.fromEntries(
|
||||
Object.entries(schema.shape).map(([key, node]) => [key, strip(node)])
|
||||
),
|
||||
} as unknown as S;
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
/**
|
||||
* Tests for the Cube runtime wrapper: default extraction and the required-key
|
||||
* check that `--use-defaults` relies on.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { z } from 'zod';
|
||||
import { Cube, Manifest } from '../src/types.js';
|
||||
import { foreignZodSchema } from './helpers/foreign-zod.js';
|
||||
|
||||
const cube = (schema: z.ZodObject<any>) =>
|
||||
new Cube(Manifest.create({ id: 'c', name: 'C', schema }), '/cubes/c', 'deploy.py');
|
||||
|
||||
describe('Cube', () => {
|
||||
it('reads id and name off the manifest', () => {
|
||||
const c = cube(z.object({}));
|
||||
|
||||
expect(c.id).toBe('c');
|
||||
expect(c.name).toBe('C');
|
||||
});
|
||||
|
||||
it('defaults its source to its own directory', () => {
|
||||
// What the loader overrides when a cube arrives from a package; a cube
|
||||
// built by hand still has to answer the question.
|
||||
expect(cube(z.object({})).source).toEqual({ type: 'dir', dir: '/cubes/c' });
|
||||
});
|
||||
|
||||
it('keeps the source it was constructed with', () => {
|
||||
const source = { type: 'package' as const, packageName: '@acme/cubes-net', dir: '/pkg/cubes' };
|
||||
const c = new Cube(
|
||||
Manifest.create({ id: 'c', name: 'C' }),
|
||||
'/pkg/cubes/c',
|
||||
'deploy.py',
|
||||
source
|
||||
);
|
||||
|
||||
expect(c.source).toBe(source);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Cube.getDefaults', () => {
|
||||
it('resolves every default when the whole schema parses', () => {
|
||||
const c = cube(
|
||||
z.object({
|
||||
PORT: z.number().default(8080),
|
||||
NAME: z.string().default('svc'),
|
||||
})
|
||||
);
|
||||
|
||||
expect(c.getDefaults()).toEqual({ PORT: 8080, NAME: 'svc' });
|
||||
});
|
||||
|
||||
it('keeps the declared defaults when one field has none', () => {
|
||||
const c = cube(
|
||||
z.object({
|
||||
SSID: z.string(),
|
||||
PRIORITY: z.number().default(10),
|
||||
HIDDEN: z.boolean().default(false),
|
||||
})
|
||||
);
|
||||
|
||||
expect(c.getDefaults()).toEqual({ PRIORITY: 10, HIDDEN: false });
|
||||
});
|
||||
|
||||
it('unwraps a default sitting under optional or nullable', () => {
|
||||
const c = cube(
|
||||
z.object({
|
||||
REQUIRED: z.string(),
|
||||
A: z.number().default(1).optional(),
|
||||
B: z.number().default(2).nullable(),
|
||||
C: z.number().optional().default(3),
|
||||
})
|
||||
);
|
||||
|
||||
expect(c.getDefaults()).toEqual({ A: 1, B: 2, C: 3 });
|
||||
});
|
||||
|
||||
it('evaluates a lazily declared default', () => {
|
||||
const c = cube(
|
||||
z.object({ REQUIRED: z.string(), TOKEN: z.string().default(() => 'generated') })
|
||||
);
|
||||
|
||||
expect(c.getDefaults()).toEqual({ TOKEN: 'generated' });
|
||||
});
|
||||
|
||||
it('omits an optional field that declares no default', () => {
|
||||
const c = cube(z.object({ REQUIRED: z.string(), MAYBE: z.string().optional() }));
|
||||
|
||||
expect(c.getDefaults()).toEqual({});
|
||||
});
|
||||
|
||||
it('returns an empty object for an empty schema', () => {
|
||||
expect(cube(z.object({})).getDefaults()).toEqual({});
|
||||
});
|
||||
|
||||
it('reads defaults off a schema built by a different copy of zod', () => {
|
||||
// The per-field fallback reads zod's internals directly. Under `instanceof`
|
||||
// a foreign schema yields no defaults at all, without erroring.
|
||||
const c = cube(
|
||||
foreignZodSchema(
|
||||
z.object({
|
||||
REQUIRED: z.string(),
|
||||
PRIORITY: z.number().default(10),
|
||||
NESTED: z.number().default(2).optional(),
|
||||
})
|
||||
)
|
||||
);
|
||||
|
||||
expect(c.getDefaults()).toEqual({ PRIORITY: 10, NESTED: 2 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('Cube.requiredKeys', () => {
|
||||
it('lists the fields with neither a default nor optionality', () => {
|
||||
const c = cube(
|
||||
z.object({
|
||||
SSID: z.string(),
|
||||
PASSWORD: z.string(),
|
||||
PRIORITY: z.number().default(10),
|
||||
NOTE: z.string().optional(),
|
||||
})
|
||||
);
|
||||
|
||||
expect(c.requiredKeys()).toEqual(['SSID', 'PASSWORD']);
|
||||
});
|
||||
|
||||
it('treats a nullable field without a default as required', () => {
|
||||
const c = cube(z.object({ MAYBE: z.string().nullable() }));
|
||||
|
||||
expect(c.requiredKeys()).toEqual(['MAYBE']);
|
||||
});
|
||||
|
||||
it('is empty when every field can fill itself in', () => {
|
||||
const c = cube(z.object({ A: z.string().default('a'), B: z.string().optional() }));
|
||||
|
||||
expect(c.requiredKeys()).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Cube.secrets', () => {
|
||||
it('is empty when the manifest declares none', () => {
|
||||
const c = cube(z.object({ PASSWORD: z.string().default('x') }));
|
||||
|
||||
expect(c.secrets).toEqual([]);
|
||||
// No name-based guessing: only what the manifest says.
|
||||
expect(c.isSecret('PASSWORD')).toBe(false);
|
||||
});
|
||||
|
||||
it('reports what the manifest declared', () => {
|
||||
const c = new Cube(
|
||||
Manifest.create({
|
||||
id: 'c',
|
||||
name: 'C',
|
||||
schema: z.object({ USER: z.string(), PASSWORD: z.string() }),
|
||||
secrets: ['PASSWORD'],
|
||||
}),
|
||||
'/cubes/c',
|
||||
'deploy.py'
|
||||
);
|
||||
|
||||
expect(c.secrets).toEqual(['PASSWORD']);
|
||||
expect(c.isSecret('PASSWORD')).toBe(true);
|
||||
expect(c.isSecret('USER')).toBe(false);
|
||||
});
|
||||
|
||||
it('defaults to an empty list on a manifest built by hand', () => {
|
||||
const c = new Cube({ id: 'c', name: 'C', schema: z.object({}) }, '/cubes/c', 'deploy.py');
|
||||
|
||||
expect(c.secrets).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -1,9 +1,9 @@
|
||||
/**
|
||||
* Tests for cubes/utils module
|
||||
* Tests for the uniqid helper
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { uniqid } from '../src/cubes/utils.js';
|
||||
import { uniqid } from '../src/utils.js';
|
||||
|
||||
describe('uniqid', () => {
|
||||
it('generates string of default length (5)', () => {
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"tsBuildInfoFile": ".tsbuildinfo",
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"lib": ["ES2020"],
|
||||
"composite": true,
|
||||
"module": "NodeNext",
|
||||
"types": ["node"]
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["coverage", "node_modules", "dist"],
|
||||
"references": []
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
globals: true,
|
||||
environment: 'node',
|
||||
include: ['src/**/*.test.ts', 'tests/**/*.test.ts'],
|
||||
coverage: {
|
||||
provider: 'v8',
|
||||
reporter: ['text', 'json-summary', 'html'],
|
||||
include: ['src/**/*.ts'],
|
||||
exclude: [
|
||||
'src/**/*.test.ts',
|
||||
// Pure re-export barrel: no logic to cover.
|
||||
'src/index.ts',
|
||||
],
|
||||
thresholds: {
|
||||
branches: 85,
|
||||
functions: 85,
|
||||
lines: 80,
|
||||
statements: 80,
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
+269
-79
@@ -6,15 +6,52 @@ A CLI tool that simplifies **pyinfra** script management and execution, providin
|
||||
|
||||
Nopy wraps pyinfra with structure, validation, and an interactive experience for managing complex infrastructure deployments. It organizes deployments into self-contained "cubes" with dependency management, schema validation, and lifecycle hooks.
|
||||
|
||||
## Features
|
||||
|
||||
- **Dependency resolution** with topological sorting
|
||||
- **Before/after hooks** for multi-cube orchestration
|
||||
- **SSH key or password authentication**
|
||||
- **Default values** with optional customization via manifest `env`
|
||||
- **Schema validation** using Zod
|
||||
- **Recursive cube directory discovery**
|
||||
- **Dry-run mode** for previewing deployments
|
||||
- **JSON output** for CI/CD integration
|
||||
- **Session history** with replay capability
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Load cubes** - Discovers and validates cubes from configured directories
|
||||
2. **Interactive prompts** - Select cubes, target host, and authentication method
|
||||
3. **Dependency resolution** - Topologically sorts cubes based on dependencies
|
||||
4. **Variable assignment** - Validates and collects configuration with schema validation
|
||||
5. **Execute hooks** - Runs before/after hooks for orchestration
|
||||
6. **Deploy** - Sequentially executes pyinfra commands
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Cubes
|
||||
|
||||
Self-contained deployment units consisting of:
|
||||
A cube is a **directory** containing two files:
|
||||
|
||||
- **Python deployment script**: `<cube-name>.deploy.py`
|
||||
- **JavaScript manifest**: `<cube-name>.manifest.mjs` defining schema, dependencies, defaults, and hooks
|
||||
- **Configuration variables**: Validated with Zod schemas
|
||||
- **JavaScript manifest**: `manifest.mjs` defining schema, dependencies, defaults, secrets, and hooks
|
||||
- **Python deployment script**: `deploy.py`, a plain pyinfra script
|
||||
|
||||
Configuration variables are declared in the manifest and validated with Zod schemas before the deployment script runs.
|
||||
|
||||
```
|
||||
cubes/
|
||||
├── .npcubes
|
||||
└── apt/
|
||||
└── install/
|
||||
├── manifest.mjs
|
||||
└── deploy.py
|
||||
```
|
||||
|
||||
Any directory holding both files is treated as a cube, so cubes can be nested as deeply as you like to group them by topic. Discovery is recursive; directories starting with `.` and `node_modules` are skipped. Additional files in the cube directory (a `README.md`, config templates, and so on) are ignored by the loader and can be referenced from the deploy script — the script runs with its cube directory as the working directory.
|
||||
|
||||
The prefixed forms `<cube-name>.manifest.mjs` and `<cube-name>.deploy.py` are also still recognized, but plain `manifest.mjs` / `deploy.py` is the current convention.
|
||||
|
||||
A cube's identity comes from the manifest's `id` field (see below). If `id` is omitted, nopy falls back to an `[id]` prefix in the manifest `name`, and finally to the directory's own name. Note that the id does not have to mirror the folder path — `cubes/network/tailscale` declares `id: 'net:tailscale'`.
|
||||
|
||||
#### Cube Manifest
|
||||
|
||||
@@ -33,18 +70,91 @@ export default cubes.Manifest({
|
||||
})
|
||||
```
|
||||
|
||||
#### Deployment Script
|
||||
|
||||
The matching `deploy.py` is a plain pyinfra script. Nopy passes each schema variable to pyinfra as `--data KEY=value`, so they are available on `host.data`:
|
||||
|
||||
```python
|
||||
from pyinfra import host
|
||||
from pyinfra.operations import apt
|
||||
|
||||
UPDATE = host.data.UPDATE
|
||||
PACKAGES = str(host.data.PACKAGES).split(' ')
|
||||
|
||||
apt.packages(
|
||||
name='Install essential packages',
|
||||
packages=['ca-certificates', 'gnupg', 'lsb-release'],
|
||||
update=UPDATE,
|
||||
_sudo=True,
|
||||
)
|
||||
|
||||
apt.packages(
|
||||
name='Install custom packages',
|
||||
packages=[p.strip() for p in PACKAGES if p],
|
||||
update=UPDATE,
|
||||
_sudo=True,
|
||||
)
|
||||
```
|
||||
|
||||
Every key defined in the manifest `schema` is guaranteed to be present on `host.data` — either from the Zod `.default()`, from `.nopyrc.json`, from a recorded session, from a dependency, or from a user prompt.
|
||||
|
||||
**Value types**: pyinfra parses `--data` values before your script sees them. `"true"` / `"false"` become booleans, numeric strings become `int`, valid JSON becomes the parsed structure, and everything else stays a string. This is why `UPDATE` can be handed straight to pyinfra's `update=` argument, while `PACKAGES` is wrapped in `str(...)` before splitting.
|
||||
|
||||
#### Variable Defaults
|
||||
|
||||
Variable defaults are defined directly in the Zod schema using `.default()`. This ensures that every cube has a predictable starting state and provides type-safe default values.
|
||||
|
||||
**Priority order (lowest to highest):**
|
||||
A variable can be set from several places in one run. Every assignment is kept, tagged with where it came from — its **origin** — and the highest-ranked origin wins.
|
||||
|
||||
1. Zod schema `.default()` values
|
||||
2. Global `env` from `.nopyrc.json`
|
||||
3. Accumulated variables from dependencies
|
||||
4. User prompts / session replay
|
||||
**Origins, lowest to highest:**
|
||||
|
||||
This allows cubes to ship with reasonable defaults while still allowing users to override them globally via `.nopyrc.json` or interactively during deployment.
|
||||
| Origin | Set by |
|
||||
| --------- | ------------------------------------------------------- |
|
||||
| `default` | the Zod schema's `.default()` |
|
||||
| `env` | the `env` block of `.nopyrc.json` |
|
||||
| `session` | a value recorded in a session file or history entry |
|
||||
| `prompt` | what the user typed |
|
||||
| `param` | a dependency spec or a `before`/`after` hook |
|
||||
|
||||
This allows cubes to ship with reasonable defaults while still allowing users to override them globally via `.nopyrc.json` or interactively during deployment. Because `env` outranks the schema, `.nopyrc.json` is also what steers a run started with `--use-defaults`, which never prompts.
|
||||
|
||||
`prompt` and `param` rarely compete: a key a dependency supplies is left out of the prompt entirely, so the user is only ever asked about the keys nothing else has set.
|
||||
|
||||
Ranking by origin rather than by arrival order is what makes replay work: a recorded value is applied *before* the cube would be prompted for, and prompting can still override it, but a `--data` value pushed in by a dependency is never clobbered by a stale recording.
|
||||
|
||||
A field declared without `.default()` has no `default` origin to fall back on. It is prompted for like any other, with an empty initial value — but a run that cannot prompt (`--use-defaults`) fails on it unless `env` or a dependency provides it.
|
||||
|
||||
#### Secrets
|
||||
|
||||
A manifest can name schema keys that hold sensitive values:
|
||||
|
||||
```javascript
|
||||
export default cubes.Manifest({
|
||||
id: 'user:add',
|
||||
name: 'Add a user account',
|
||||
secrets: ['PASSWORD'],
|
||||
schema: z.object({
|
||||
USERNAME: z.string().describe('Username for the new account').default('deploy'),
|
||||
PASSWORD: z.string().describe('Password for the new user account').default('changeme'),
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
Every entry must be a key of `schema`; naming anything else is a manifest error and aborts the run, so a typo fails loudly instead of silently leaving a value unprotected.
|
||||
|
||||
Declaring a key a secret changes three things:
|
||||
|
||||
- **It is never written to a session file or to the history.** Everything else the run settled on is recorded — including values that came from a `.default()` — but declared secrets are left out.
|
||||
- **It is masked wherever a command or a plan is printed** — `--dry-run`, `--print-only`, and the debug log all show `********` in place of the value, in the variable list *and* in the `pyinfra` command line above it. The SSH password passed via `--password` is masked the same way, whether or not any cube declares secrets.
|
||||
- **It is re-prompted on replay**, since there is nothing recorded to replay from (see [Session Recording and Replay](#session-recording-and-replay)).
|
||||
|
||||
Nopy does not guess. A key called `PASSWORD` in a manifest that declares no `secrets` is treated as an ordinary variable — recorded, and printed in the clear.
|
||||
|
||||
Three limits are worth knowing, because `secrets` keeps a value out of the files nopy writes and nothing more:
|
||||
|
||||
- **It is on the command line.** pyinfra takes its data as `--data KEY=value`, so the real value is visible in `ps` for as long as the deployment runs. Masking covers nopy's own output, not the process table.
|
||||
- **The prompt shows it.** The variable form displays and pre-fills what it is asking about, so a secret is on screen while it is being entered or confirmed.
|
||||
- **A `.default()` is not protected.** A default lives in the manifest, in plain text, wherever the manifest is checked in. Give a secret a placeholder default like `changeme` if it needs one at all, never a real credential.
|
||||
|
||||
### Configuration
|
||||
|
||||
@@ -54,16 +164,28 @@ Uses `.nopyrc.json` files (project-level or home directory) containing:
|
||||
{
|
||||
"hosts": ["host1.example.com", "host2.example.com"],
|
||||
"cubeDirs": ["./cubes", "../shared-cubes"],
|
||||
"cubePackages": ["@bitsquare/cubes-core"],
|
||||
"env": {
|
||||
"SHARED_VAR": "value"
|
||||
},
|
||||
"log": {
|
||||
"verbosity": "info",
|
||||
"debug": false
|
||||
},
|
||||
"history": {
|
||||
"maxSessions": 10,
|
||||
"autoSave": true
|
||||
},
|
||||
"execution": {
|
||||
"continueOnError": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`history` controls automatic session recording (see [Deployment History](#deployment-history)), and `execution.continueOnError` sets the default for `--continue-on-error`.
|
||||
|
||||
`cubeDirs` holds paths, `cubePackages` holds installed npm packages that ship cubes — see [Cube Discovery](#cube-discovery) below and [CUBE-BUNDLES.md](docs/CUBE-BUNDLES.md) for publishing your own. Both are additive, and both resolve relative to the config file that named them, not to the working directory: a `.nopyrc.json` two levels up may name a package that only exists in *its* `node_modules`.
|
||||
|
||||
#### Logging Configuration
|
||||
|
||||
Control pyinfra output verbosity and debug information using the `log` configuration object:
|
||||
@@ -84,39 +206,6 @@ Control pyinfra output verbosity and debug information using the `log` configura
|
||||
| `false` | (none) | No debug logs (default) | Normal operation |
|
||||
| `true` | `--debug` | Enable pyinfra debug logs | Deep debugging of pyinfra internals |
|
||||
|
||||
**Examples:**
|
||||
|
||||
Basic troubleshooting:
|
||||
|
||||
```json
|
||||
{
|
||||
"log": {
|
||||
"verbosity": "info"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Debug command failures:
|
||||
|
||||
```json
|
||||
{
|
||||
"log": {
|
||||
"verbosity": "trace"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Deep debugging with pyinfra internals:
|
||||
|
||||
```json
|
||||
{
|
||||
"log": {
|
||||
"verbosity": "trace",
|
||||
"debug": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Recommendation:** Start with `"info"` for typical troubleshooting, use `"trace"` when investigating command failures, and enable `debug: true` only when debugging pyinfra itself.
|
||||
|
||||
### Session Recording and Replay
|
||||
@@ -166,12 +255,16 @@ Sessions are stored in `.nopysession.json` files with the following structure:
|
||||
|
||||
**Structure Details:**
|
||||
|
||||
- **`cubes`**: Array of cubes with only cube-specific variables (not global env vars)
|
||||
- **`env`**: Global environment variables shared across cubes (like in `.nopyrc.json`)
|
||||
- **`cubes`**: Array of cubes with the variable values that cube ran with
|
||||
- **`env`**: The `env` block of `.nopyrc.json` as it stood at record time, kept for reference
|
||||
- **`hosts`**: Array of target hosts
|
||||
- **`auth`**: Authentication configuration (passwords are never stored)
|
||||
|
||||
**Security Note**: Passwords are never stored in session files. If a session uses password authentication, you'll be prompted for the password during replay.
|
||||
**What is recorded:** every value each cube settled on, regardless of where it came from — a value the user typed, one inherited from `.nopyrc.json` `env`, one a dependency supplied, and one that fell through to the schema's `.default()` are all written out the same way. A session is therefore a full snapshot rather than a diff, and a `--use-defaults` run produces a session with real values in it instead of an empty one.
|
||||
|
||||
The consequence is that replay is faithful rather than re-derived: the recorded value outranks the current `.nopyrc.json` `env` and the current schema default, so editing either one does not silently change what a replay does. To pick up a new default, record a fresh session.
|
||||
|
||||
**Security Note**: Passwords are never stored in session files. This covers both the SSH password — a session records the auth *method* and username, never the credential — and any schema key a cube's manifest lists under [`secrets`](#secrets). Both are re-prompted on replay.
|
||||
|
||||
#### Recording a Session
|
||||
|
||||
@@ -193,12 +286,48 @@ nopy install --load-session my-deployment.nopysession.json
|
||||
# Only password authentication will prompt for credentials
|
||||
```
|
||||
|
||||
A replay runs straight through without asking anything, with three exceptions. Password authentication always re-prompts. A session with no recorded host falls back to the host picker. And a cube is re-prompted for its declared secrets, plus for any required variable the session has no value for — which happens when the cube's schema has gained a field since the session was written.
|
||||
|
||||
Those re-prompts are what a session cannot supply, so `--use-defaults` cannot paper over them: combining `-D` with a replay that needs either fails with a message naming the keys rather than deploying with a placeholder. Put the values under `env` in `.nopyrc.json` to make such a replay unattended.
|
||||
|
||||
### Cube Discovery
|
||||
|
||||
Nopy searches for cubes in:
|
||||
|
||||
1. Directories specified in `.nopyrc.json` `cubeDirs`
|
||||
2. Directories containing a `.npcubes` marker file (searching upwards from current directory)
|
||||
2. Cube directories of every package listed in `.nopyrc.json` `cubePackages`
|
||||
3. Directories containing a `.npcubes` marker file (searching upwards from current directory)
|
||||
|
||||
All three are unioned and scanned the same way. A directory is a cube when it holds both a manifest (`manifest.mjs` or `*.manifest.mjs`) and a deploy script (`deploy.py` or `*.deploy.py`); dotted directories and `node_modules` are skipped during the scan.
|
||||
|
||||
#### Cube packages
|
||||
|
||||
A cube package is an ordinary npm package that ships cube directories and points at them from its own `package.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@bitsquare/cubes-core",
|
||||
"nopy": { "cubes": ["./cubes"] }
|
||||
}
|
||||
```
|
||||
|
||||
Install it and name it — nothing needs linking or copying:
|
||||
|
||||
```sh
|
||||
pnpm add -D @bitsquare/cubes-core
|
||||
```
|
||||
|
||||
```json
|
||||
{ "cubePackages": ["@bitsquare/cubes-core"] }
|
||||
```
|
||||
|
||||
Naming a package is a statement that cubes are expected from it, so anything wrong is an error that aborts the run rather than a silent skip: the package is not installed, it declares no `nopy.cubes`, or an entry points at a directory that does not exist or lies outside the package.
|
||||
|
||||
#### Ids are claimed globally
|
||||
|
||||
A cube id such as `apt:essentials` is claimed across every source at once, not per directory or per package. Two cubes with the same id abort the run with an error naming both and where each came from. There is no precedence rule and no shadowing — a local cube does not quietly win over a packaged one, in either direction. Prefix your own cubes distinctly if you point `cubeDirs` at a local tree alongside an installed bundle.
|
||||
|
||||
Writing cubes to publish is covered in [CUBE-BUNDLES.md](docs/CUBE-BUNDLES.md).
|
||||
|
||||
## Command Line Usage
|
||||
|
||||
@@ -253,6 +382,36 @@ nopy install --use-defaults
|
||||
nopy install -D
|
||||
```
|
||||
|
||||
Skips the per-cube variable form. Every variable is taken from the sources that
|
||||
need no interaction — the Zod `.default()`, `env` in `.nopyrc.json`, and values
|
||||
handed over by a dependency or a hook — which is what makes `.nopyrc.json` the
|
||||
place to configure an unattended run.
|
||||
|
||||
Cube selection, host and authentication are still asked for; there is nowhere
|
||||
else for them to come from. Pair `-D` with `-K` to skip the auth question too,
|
||||
or with `-R` / `-H` / `-l`, which supply all three from the recorded session.
|
||||
|
||||
A cube whose schema declares a field with **no** `.default()` cannot be filled in
|
||||
this way, so the run stops before anything is deployed rather than passing the
|
||||
variable as empty:
|
||||
|
||||
```
|
||||
Error: Cube "net:wifi:connection" cannot run with --use-defaults: SSID, PASSWORD
|
||||
have no default values. Set them under "env" in .nopyrc.json, pass them from a
|
||||
dependency, or drop --use-defaults to be prompted.
|
||||
```
|
||||
|
||||
Pairing `-D` with a replay fails the same way when the replay would have to ask
|
||||
something — a declared [secret](#secrets), which is never recorded, or a required
|
||||
variable the session has no value for. Both are the sources `-D` has no substitute
|
||||
for, so it stops rather than deploying a placeholder:
|
||||
|
||||
```
|
||||
Error: Cube "user:add" cannot be replayed with --use-defaults: PASSWORD would
|
||||
have to be entered. Secrets are never recorded in a session. Replay without
|
||||
--use-defaults, or set the values under "env" in .nopyrc.json.
|
||||
```
|
||||
|
||||
**Use SSH key authentication**:
|
||||
|
||||
```bash
|
||||
@@ -264,11 +423,23 @@ nopy install -K
|
||||
**Repeat last run**:
|
||||
|
||||
```bash
|
||||
nopy install --repeat-last-run
|
||||
nopy install --repeat-last
|
||||
# or
|
||||
nopy install -R
|
||||
```
|
||||
|
||||
Every deployment is automatically recorded to a `.nopy.history.json` file in the current working directory, so the last run is always available to `-R` without having to pass `--save-session` first. The default retention is the 10 most recent sessions (configurable via `history.maxSessions`); use `nopy history` to list them and `nopy install -H <id>` to replay any one of them — see [Deployment History](#deployment-history).
|
||||
|
||||
The recording happens before the deploy commands run, so a **failed** deployment is recorded too — `-R` is the quick way to retry one after fixing the cause. Replaying a session with `-R` or `-H` does not itself create a new entry, so repeating never pushes the original run out of the list.
|
||||
|
||||
A run is *not* recorded when:
|
||||
|
||||
- `--dry-run` or `--no-history` is passed
|
||||
- No cubes were selected, so there was nothing to deploy
|
||||
- `history.autoSave` is set to `false` in `.nopyrc.json`
|
||||
|
||||
Because the history file is resolved against the current working directory, each project keeps its own history — running nopy from a different directory will not find the previous run. As with session files, passwords are never stored and are re-prompted on replay.
|
||||
|
||||
**Save session for replay**:
|
||||
|
||||
```bash
|
||||
@@ -302,14 +473,6 @@ nopy install --dry-run
|
||||
|
||||
Shows the execution plan including commands, environment variables, and targets without running anything. Sensitive data is masked in output.
|
||||
|
||||
**Parallel execution**:
|
||||
|
||||
```bash
|
||||
nopy install --parallel
|
||||
```
|
||||
|
||||
Executes independent cubes in parallel using a dependency graph. Cubes are grouped into execution stages, with a default concurrency limit of 4.
|
||||
|
||||
**JSON output (for CI/CD)**:
|
||||
|
||||
```bash
|
||||
@@ -323,17 +486,64 @@ Machine-readable JSON output for scripting and CI/CD integration.
|
||||
|
||||
```bash
|
||||
nopy install --continue-on-error
|
||||
# or
|
||||
nopy install -c
|
||||
```
|
||||
|
||||
Continue deploying remaining cubes even if one fails.
|
||||
|
||||
**View deployment history**:
|
||||
**Default behaviour (fail-fast)**: without this flag, nopy stops at the first cube that fails. Cubes are deployed sequentially in dependency order, so the failing cube's output is the last thing you see — every cube still queued behind it is skipped entirely and is never attempted.
|
||||
|
||||
This is deliberate: because cubes are topologically sorted, a cube that fails is often a dependency of the ones after it, and continuing would deploy them onto a half-configured host.
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
||||
- **Cubes that already succeeded are not rolled back.** The host is left in a partial state — the cubes before the failure are applied, the rest are not. Fix the cause and re-run; well-written cubes are idempotent, so re-applying the earlier ones is normally harmless.
|
||||
- **Skipped cubes are not reported as failed.** They are simply absent from the results, so a summary of "3 successful, 1 failed" out of 6 cubes means the remaining 2 were never run.
|
||||
|
||||
Either way, the command exits with code `1` if any cube failed, which is what CI picks up. Use `--continue-on-error` when your cubes are genuinely independent and you would rather collect every failure in one run than stop at the first.
|
||||
|
||||
The default can be flipped for a project by setting `execution.continueOnError` in `.nopyrc.json`; the CLI flag takes precedence over it.
|
||||
|
||||
#### Deployment History
|
||||
|
||||
```bash
|
||||
nopy history # List recent deployments
|
||||
nopy history --json # Same list as JSON, including each recorded session
|
||||
nopy install -H <id> # Replay a specific deployment by ID
|
||||
nopy clear-history # Delete all recorded sessions
|
||||
```
|
||||
|
||||
History is what makes [Repeat last run](#basic-commands) work, but it holds more than just the last deployment: every recorded run stays replayable until newer runs push it out. `nopy history` (alias `nopy h`) lists them newest first, with `→` marking the entry that `-R` would replay:
|
||||
|
||||
```
|
||||
Session History:
|
||||
|
||||
→ [1] 07/26/2026, 14:32 - apt:install, net:tailscale → root@web-01
|
||||
ID: mdk3n1qx4a2fh
|
||||
[2] 07/26/2026, 09:05 - apt:install → root@web-01
|
||||
ID: mdk0zzp8b71cq
|
||||
|
||||
Total: 2 session(s)
|
||||
```
|
||||
|
||||
Each entry records the selected cubes together with every variable value they ran with, the target hosts, the authentication method, and the username — never the password, and never a key the manifest declared a [secret](#secrets). Pass an ID to `-H` to run that exact combination again:
|
||||
|
||||
```bash
|
||||
nopy install -H mdk0zzp8b71cq
|
||||
```
|
||||
|
||||
A replay is non-interactive: cube selection, host, and variable values all come from the entry, so nopy runs straight through without asking anything. It asks only for what the entry cannot hold — the password under password authentication, and any declared secret — plus the host picker when the entry recorded none.
|
||||
|
||||
Two things are worth knowing before relying on an older entry:
|
||||
|
||||
- **Recorded values win over the current configuration.** The entry is a snapshot of everything the run settled on, so editing a cube's `.default()` or the `env` block of `.nopyrc.json` afterwards does not change what the replay does. A variable the schema has gained *since* the entry was written has nothing recorded: if it has a `.default()` the replay quietly takes it, and if it is required the replay prompts for it.
|
||||
- **A replay fails if a cube no longer exists.** Renaming or deleting a cube id makes every history entry that referenced it unreplayable: nopy logs `Cube from session not found` and then aborts with `Cube not found: <id>`.
|
||||
|
||||
The history lives in `.nopy.history.json` in the working directory and uses the same structure as a session file, so trimming the array by hand is a perfectly good way to prune it. It does contain the variable values a run used, which is why it is listed in this repository's `.gitignore` — treat it like any other file holding deployment configuration. A corrupt or unreadable history file is treated as empty rather than raising an error, which looks exactly like a project that has never been deployed from.
|
||||
|
||||
For a run you want to keep indefinitely, don't rely on history — it rotates. Use `--save-session` to write it to a file you control (see [Session Recording and Replay](#session-recording-and-replay)).
|
||||
|
||||
### Development
|
||||
|
||||
**Run without building**:
|
||||
@@ -348,32 +558,12 @@ npm run nopy
|
||||
npm run debug
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Load cubes** - Discovers and validates cubes from configured directories
|
||||
2. **Interactive prompts** - Select cubes, target host, and authentication method
|
||||
3. **Dependency resolution** - Topologically sorts cubes based on dependencies
|
||||
4. **Variable assignment** - Validates and collects configuration with schema validation
|
||||
5. **Execute hooks** - Runs before/after hooks for orchestration
|
||||
6. **Deploy** - Sequentially executes pyinfra commands
|
||||
|
||||
## Features
|
||||
|
||||
- **Dependency resolution** with topological sorting
|
||||
- **Parallel execution** of independent cubes in stages
|
||||
- **Before/after hooks** for multi-cube orchestration
|
||||
- **SSH key or password authentication**
|
||||
- **Default values** with optional customization via manifest `env`
|
||||
- **Schema validation** using Zod
|
||||
- **Recursive cube directory discovery**
|
||||
- **Dry-run mode** for previewing deployments
|
||||
- **JSON output** for CI/CD integration
|
||||
- **Session history** with replay capability
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Cube Hooks](docs/HOOKS.md) - Lifecycle hooks for dynamic orchestration
|
||||
- [Cube Bundles](docs/CUBE-BUNDLES.md) - Distributing cubes as npm packages
|
||||
- [Session Format](docs/SESSION_FORMAT.md) - Internal JSON/MJS session structure
|
||||
- [API Reference](docs/API.md) - Types and exported functions
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
|
||||
export default cubes.Manifest({
|
||||
name: '[apt-all] Test dependencies',
|
||||
dependencies: () => ['apt/more'],
|
||||
name: '[test:apt-all] Test dependencies',
|
||||
dependencies: () => ['test:apt-more'],
|
||||
});
|
||||
|
||||
@@ -2,7 +2,7 @@ import { cubes } from '@bitsquare/nopy';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default cubes.Manifest({
|
||||
name: '[apt:essentials] Install essential packages',
|
||||
name: '[test:apt-essentials] Install essential packages',
|
||||
dependencies: () => [],
|
||||
schema: z.object({
|
||||
UPDATE: z.boolean().default(false),
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
|
||||
export default cubes.Manifest({
|
||||
name: '[apt-more] Test dependencies',
|
||||
dependencies: () => [['apt/essentials']],
|
||||
name: '[test:apt-more] Test dependencies',
|
||||
dependencies: () => [['test:apt-essentials']],
|
||||
});
|
||||
|
||||
+92
-30
@@ -41,7 +41,6 @@ const result = await nopy({
|
||||
| `saveSession` | `string` | - | Path to save session file |
|
||||
| `loadSession` | `string` | - | Path to load session for replay |
|
||||
| `dryRun` | `boolean` | `false` | Show execution plan without running |
|
||||
| `parallel` | `boolean` | `false` | Execute independent cubes in parallel |
|
||||
| `continueOnError` | `boolean` | `false` | Continue after failures |
|
||||
| `jsonOutput` | `boolean` | `false` | Output results as JSON |
|
||||
|
||||
@@ -66,6 +65,20 @@ interface NopyResult {
|
||||
|
||||
The cubes module provides types and functions for working with deployment units.
|
||||
|
||||
The authoring half of it — `Manifest`, `Cube`, `Hook`, `uniqid` and the rest —
|
||||
actually lives in **[`@bitsquare/nopy-cube`](../../nopy-cube)**, a package with
|
||||
no CLI and no dependency other than zod. `@bitsquare/nopy` re-exports all of it,
|
||||
so both of these work:
|
||||
|
||||
```javascript
|
||||
import { Manifest } from '@bitsquare/nopy-cube'; // in a manifest.mjs — prefer this
|
||||
import { cubes } from '@bitsquare/nopy'; // cubes.Manifest — still supported
|
||||
```
|
||||
|
||||
Import from `nopy-cube` in a cube bundle you intend to publish: it lets the
|
||||
bundle depend on the authoring types without pulling the whole CLI in as a
|
||||
dependency. See [CUBE-BUNDLES.md](CUBE-BUNDLES.md).
|
||||
|
||||
### Types
|
||||
|
||||
#### `Cube<Schema>`
|
||||
@@ -77,6 +90,7 @@ interface Cube<Schema extends z.AnyZodObject = z.AnyZodObject> {
|
||||
key: string; // Unique identifier
|
||||
name: string; // Human-readable name
|
||||
dir: string; // Absolute path to cube directory
|
||||
source: CubeSource; // Where it was discovered
|
||||
dependencies: string[];
|
||||
schema: Schema;
|
||||
defaults: () => z.infer<Schema>;
|
||||
@@ -85,9 +99,21 @@ interface Cube<Schema extends z.AnyZodObject = z.AnyZodObject> {
|
||||
}
|
||||
```
|
||||
|
||||
#### `CubeSource`
|
||||
|
||||
Where a cube came from. Carried so that a duplicate-id error can name the origin
|
||||
of each claimant, which is the difference between a usable error message and a
|
||||
puzzle when the collision is between a local tree and an installed bundle.
|
||||
|
||||
```typescript
|
||||
type CubeSource =
|
||||
| { type: 'dir'; dir: string }
|
||||
| { type: 'package'; packageName: string; dir: string };
|
||||
```
|
||||
|
||||
#### `Manifest<Schema>`
|
||||
|
||||
Cube manifest (used in `*.manifest.mjs` files).
|
||||
Cube manifest (used in `manifest.mjs` files).
|
||||
|
||||
```typescript
|
||||
interface Manifest<Schema extends z.AnyZodObject = z.AnyZodObject> {
|
||||
@@ -125,7 +151,9 @@ interface HookContext {
|
||||
|
||||
#### `loadCubes()`
|
||||
|
||||
Loads all cubes from discovered cube directories.
|
||||
Loads all cubes from discovered cube directories — `cubeDirs`, the directories
|
||||
declared by every package in `cubePackages`, and any ancestor directory holding a
|
||||
`.npcubes` marker.
|
||||
|
||||
```typescript
|
||||
const { cubes, errors } = await loadCubes();
|
||||
@@ -140,6 +168,27 @@ interface LoadResult {
|
||||
}
|
||||
```
|
||||
|
||||
`errors` is non-empty for a duplicate id, a manifest that fails to load, a
|
||||
package in `cubePackages` that is not installed or declares no cubes, and a
|
||||
`nopy.cubes` entry that is missing or points outside its package. Any of them
|
||||
aborts the run — none is a silent skip.
|
||||
|
||||
#### `resolveCubePackages(refs)`
|
||||
|
||||
Resolves `CubePackageRef[]` to installed packages and their cube directories.
|
||||
Called by `loadCubes()`; exported because the resolution failures are worth
|
||||
testing on their own.
|
||||
|
||||
```typescript
|
||||
const { packages, errors } = resolveCubePackages(config.cubePackages);
|
||||
|
||||
interface CubePackage {
|
||||
name: string; // the name it was requested under
|
||||
root: string; // absolute path to the package root
|
||||
dirs: string[]; // absolute paths from its `nopy.cubes` field
|
||||
}
|
||||
```
|
||||
|
||||
#### `resolveDependencies(cubes, selectedCubeNames)`
|
||||
|
||||
Resolves all transitive dependencies for selected cubes.
|
||||
@@ -160,23 +209,14 @@ const order = resolveDependencies(cubes, ['apt-all']);
|
||||
|
||||
**Throws:** `Error` if cube not found or circular dependency detected
|
||||
|
||||
#### `buildExecutionStages(cubes, selectedCubeNames)`
|
||||
|
||||
Groups cubes into stages for parallel execution.
|
||||
|
||||
```typescript
|
||||
const stages = buildExecutionStages(cubes, ['apt-all', 'docker']);
|
||||
// Returns: [['apt:essentials'], ['apt-more', 'docker'], ['apt-all']]
|
||||
```
|
||||
|
||||
**Returns:** `string[][]` - Array of stages
|
||||
|
||||
#### `createManifest(options)`
|
||||
#### `cubes.Manifest(options)`
|
||||
|
||||
Factory function for creating cube manifests.
|
||||
|
||||
```typescript
|
||||
export default createManifest({
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
|
||||
export default cubes.Manifest({
|
||||
name: 'My Cube',
|
||||
dependencies: () => [['apt:essentials']],
|
||||
schema: z.object({
|
||||
@@ -185,6 +225,8 @@ export default createManifest({
|
||||
});
|
||||
```
|
||||
|
||||
`createManifest` and `manifest` are exported as equivalent aliases; `cubes.Manifest` is the documented form.
|
||||
|
||||
#### `uniqid(length?)`
|
||||
|
||||
Generates a random alphanumeric string.
|
||||
@@ -239,8 +281,6 @@ Options for deployment execution.
|
||||
|
||||
```typescript
|
||||
interface ExecutionOptions {
|
||||
parallel?: boolean;
|
||||
concurrency?: number;
|
||||
continueOnError?: boolean;
|
||||
dryRun?: boolean;
|
||||
onProgress?: (result: ExecutionResult, completed: number, total: number) => void;
|
||||
@@ -252,12 +292,11 @@ interface ExecutionOptions {
|
||||
|
||||
#### `executeDeployCalls(calls, options?)`
|
||||
|
||||
Executes an array of deployment calls.
|
||||
Executes an array of deployment calls sequentially, in the order they were built.
|
||||
|
||||
```typescript
|
||||
const results = await executeDeployCalls(calls, {
|
||||
parallel: true,
|
||||
concurrency: 4,
|
||||
continueOnError: false,
|
||||
onProgress: (result, completed, total) => {
|
||||
console.log(`${completed}/${total}`);
|
||||
},
|
||||
@@ -463,11 +502,31 @@ Configuration file structure.
|
||||
interface NopyConfig {
|
||||
hosts: string[];
|
||||
cubeDirs: string[];
|
||||
cubePackages: CubePackageRef[];
|
||||
env: EnvConfig;
|
||||
log?: LogConfig;
|
||||
}
|
||||
```
|
||||
|
||||
#### `CubePackageRef`
|
||||
|
||||
A package named in `cubePackages`, paired with where it was named. In the config
|
||||
file an entry is just a string (`"@bitsquare/cubes-core"`); `loadConfig()`
|
||||
normalises it.
|
||||
|
||||
```typescript
|
||||
interface CubePackageRef {
|
||||
/** The package name, as written in the config. */
|
||||
spec: string;
|
||||
/** Directory of the config file that named it — resolution starts here. */
|
||||
from: string;
|
||||
}
|
||||
```
|
||||
|
||||
`from` is what makes a package named in a parent config resolve against *that*
|
||||
config's `node_modules`, not the working directory's. It is the same problem
|
||||
`PATH_PROPERTIES` solves for relative `cubeDirs`.
|
||||
|
||||
#### `LogConfig`
|
||||
|
||||
Logging configuration.
|
||||
@@ -581,9 +640,6 @@ nopy install -l ./my-session.json
|
||||
# Dry run
|
||||
nopy install -n
|
||||
|
||||
# Parallel execution
|
||||
nopy install -p
|
||||
|
||||
# JSON output
|
||||
nopy install -j
|
||||
|
||||
@@ -597,21 +653,27 @@ nopy install -c
|
||||
|
||||
### File Structure
|
||||
|
||||
A cube is a directory containing both a `manifest.mjs` and a `deploy.py`:
|
||||
|
||||
```
|
||||
cubes/
|
||||
└── my-cube/
|
||||
├── my-cube.manifest.mjs
|
||||
└── my-cube.deploy.py
|
||||
├── manifest.mjs
|
||||
└── deploy.py
|
||||
```
|
||||
|
||||
Cube directories may be nested for grouping (`cubes/apt/install/`), and any extra files alongside the pair are available to the deploy script via relative paths.
|
||||
|
||||
The prefixed forms `<cube-name>.manifest.mjs` and `<cube-name>.deploy.py` are still recognized for backwards compatibility.
|
||||
|
||||
### Manifest Example
|
||||
|
||||
```javascript
|
||||
// my-cube.manifest.mjs
|
||||
import { createManifest } from '@bitsquare/nopy';
|
||||
// manifest.mjs
|
||||
import { cubes } from '@bitsquare/nopy';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default createManifest({
|
||||
export default cubes.Manifest({
|
||||
name: 'My Cube',
|
||||
dependencies: () => [['apt:essentials']],
|
||||
schema: z.object({
|
||||
@@ -634,7 +696,7 @@ export default createManifest({
|
||||
### Deploy Script Example
|
||||
|
||||
```python
|
||||
# my-cube.deploy.py
|
||||
# deploy.py
|
||||
from pyinfra import host
|
||||
from pyinfra.operations import apt, server
|
||||
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
# Cube bundles
|
||||
|
||||
How to package cubes as an npm package so other projects can install them, and
|
||||
what changes once a cube lives in `node_modules` instead of in your own tree.
|
||||
|
||||
If you only want to *use* a published bundle, you need one line of config:
|
||||
|
||||
```json
|
||||
{ "cubePackages": ["@bitsquare/cubes-core"] }
|
||||
```
|
||||
|
||||
The rest of this document is for writing one.
|
||||
|
||||
- [What a bundle is](#what-a-bundle-is)
|
||||
- [The package manifest](#the-package-manifest)
|
||||
- [Writing the cubes](#writing-the-cubes)
|
||||
- [Ids are claimed globally](#ids-are-claimed-globally)
|
||||
- [An installed bundle is read-only](#an-installed-bundle-is-read-only)
|
||||
- [How resolution actually works](#how-resolution-actually-works)
|
||||
- [Publishing](#publishing)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
## What a bundle is
|
||||
|
||||
An ordinary npm package that ships cube directories and points at them from its
|
||||
own `package.json`. There is no build step, no plugin API and no entry point —
|
||||
nopy reads the directories off disk and imports each `manifest.mjs` directly.
|
||||
|
||||
```
|
||||
@acme/cubes-web
|
||||
├── package.json nopy.cubes → ["./cubes"]
|
||||
├── README.md
|
||||
└── cubes/
|
||||
├── nginx/
|
||||
│ ├── manifest.mjs
|
||||
│ └── deploy.py
|
||||
└── certbot/
|
||||
├── manifest.mjs
|
||||
└── deploy.py
|
||||
```
|
||||
|
||||
`@bitsquare/cubes-core` in this repository is the worked example, and is consumed
|
||||
by this repository through exactly the mechanism described here — it is not
|
||||
special-cased.
|
||||
|
||||
## The package manifest
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@acme/cubes-web",
|
||||
"version": "1.0.0",
|
||||
"type": "module",
|
||||
"nopy": { "cubes": ["./cubes"] },
|
||||
"files": ["cubes", "!cubes/**/*.log", "README.md", "LICENSE"],
|
||||
"publishConfig": { "access": "public" },
|
||||
"dependencies": {
|
||||
"@bitsquare/nopy-cube": "^1.0.0",
|
||||
"zod": "^4.4.3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`nopy.cubes`** is the only field nopy requires. It is an array of directories,
|
||||
relative to the package root, each scanned recursively for cubes. Several
|
||||
entries are fine; a single `["./cubes"]` is the norm. Every entry must exist and
|
||||
must stay inside the package — a path escaping the root is refused, not resolved.
|
||||
|
||||
**`type: "module"`** matters: manifests are ESM. Without it a `manifest.mjs` still
|
||||
loads (the extension carries the day), but anything it imports relatively will
|
||||
not behave the way you expect.
|
||||
|
||||
**`files`** decides the tarball. Note the negation: a cube that has been run
|
||||
leaves a `pyinfra-debug.log` next to its `deploy.py`, and `.gitignore` has no
|
||||
effect on what npm packs. Check with `npm pack --dry-run` before publishing.
|
||||
|
||||
**Dependencies** are `@bitsquare/nopy-cube` and `zod`, both real dependencies
|
||||
rather than peers — a bundle is a leaf, and the copies it gets are the copies its
|
||||
manifests use. Do **not** depend on `@bitsquare/nopy`: the CLI is what installs
|
||||
your bundle, not the other way round, and depending on it invites two copies of
|
||||
the same code into one tree.
|
||||
|
||||
## Writing the cubes
|
||||
|
||||
A cube directory holds a manifest (`manifest.mjs` or `*.manifest.mjs`) and a
|
||||
deploy script (`deploy.py` or `*.deploy.py`). Anything else in the directory is
|
||||
invisible to the loader but readable from the script, which runs with the cube
|
||||
directory as its working directory.
|
||||
|
||||
```javascript
|
||||
// cubes/nginx/manifest.mjs
|
||||
import { Manifest } from '@bitsquare/nopy-cube';
|
||||
import { z } from 'zod';
|
||||
|
||||
export default Manifest({
|
||||
id: 'web:nginx',
|
||||
name: 'Install and configure nginx',
|
||||
dependencies: () => ['apt:essentials'],
|
||||
secrets: ['TLS_KEY'],
|
||||
schema: z.object({
|
||||
SERVER_NAME: z.string().describe('Server name').default('example.com'),
|
||||
TLS_KEY: z.string().describe('TLS private key (PEM)').default(''),
|
||||
HTTP2: z.boolean().describe('Enable HTTP/2').default(true),
|
||||
}),
|
||||
});
|
||||
```
|
||||
|
||||
```python
|
||||
# cubes/nginx/deploy.py
|
||||
from pyinfra import host
|
||||
from pyinfra.operations import apt, files
|
||||
|
||||
SERVER_NAME = host.data.SERVER_NAME
|
||||
|
||||
apt.packages(name='Install nginx', packages=['nginx'], _sudo=True)
|
||||
```
|
||||
|
||||
Import **`@bitsquare/nopy-cube`**, not `@bitsquare/nopy`. It is types and a
|
||||
factory with zod as its only peer — no CLI, no prompts, no process spawning — so
|
||||
your bundle stays a leaf. (`@bitsquare/nopy` re-exports the same surface as
|
||||
`cubes.Manifest`, which is what older manifests use. It still works; it just
|
||||
drags the CLI into your dependency graph if you declare it.)
|
||||
|
||||
Four things the schema is load-bearing for:
|
||||
|
||||
- **`.describe()` is the prompt label.** A field without one prompts with its raw
|
||||
key.
|
||||
- **`.default()` makes the field optional.** A field with no default is required,
|
||||
and is re-prompted on replay if a session has no value for it.
|
||||
- **Every schema key reaches pyinfra** as `--data KEY=value`, so `host.data.KEY`
|
||||
is always defined. pyinfra parses the values itself: `"true"` arrives as a
|
||||
bool, `"8080"` as an int.
|
||||
- **`secrets` names keys whose values must not be persisted.** They are excluded
|
||||
from session files and history, masked wherever a command is printed, and
|
||||
re-prompted on replay. Naming a key that is not in the schema is a load error.
|
||||
A secret is still visible in `ps` while pyinfra runs — masking covers nopy's
|
||||
own output, not the process table — so treat it as protection against writing
|
||||
credentials to disk, not as protection against a shared host.
|
||||
|
||||
`dependencies` is a function of the *collected* variables, so it can branch on
|
||||
what the user actually answered, and it may pass parameters:
|
||||
|
||||
```javascript
|
||||
dependencies: (v) => (v.HTTP2 ? ['apt:essentials', ['web:tls', { MODE: 'strict' }]] : []),
|
||||
```
|
||||
|
||||
`before` / `after` hooks get a context whose `exec(id, vars)` pulls in any cube
|
||||
by id, declared dependency or not. See [HOOKS.md](HOOKS.md).
|
||||
|
||||
## Ids are claimed globally
|
||||
|
||||
An id is claimed across every source at once — `cubeDirs`, `.npcubes` trees and
|
||||
every installed bundle share one flat namespace. Two cubes claiming the same id
|
||||
abort the run with an error naming both and where each came from.
|
||||
|
||||
There is no precedence and no shadowing, deliberately, in either direction: a
|
||||
local cube does not quietly win over a packaged one, and installing a second
|
||||
bundle cannot silently change what an existing id deploys. Overriding a cube from
|
||||
a bundle is not a supported operation; fork the cube under your own id instead.
|
||||
|
||||
So prefix distinctly. `@acme/cubes-web` claiming `nginx` is asking for trouble the
|
||||
first time someone installs a second bundle; `web:nginx` is not. Ids need not
|
||||
mirror the directory layout — `cubes/network/tailscale` declares `net:tailscale`
|
||||
— so the prefix is free.
|
||||
|
||||
An id is also the session key. Renaming one silently invalidates every recorded
|
||||
session that used it, so treat a rename as a breaking change of the bundle.
|
||||
|
||||
## An installed bundle is read-only
|
||||
|
||||
Under pnpm, installed files are **hardlinked into a global store shared by every
|
||||
project on the machine**. A cube that writes next to its own `deploy.py` does not
|
||||
just dirty one `node_modules` — it corrupts that store for every other project.
|
||||
|
||||
Write to `/tmp`, to a path the user configured, or to the remote host. Never to
|
||||
the cube's own directory. Files the cube needs to *read* (templates, config
|
||||
fragments, systemd units) are fine and are exactly what the cube directory is for
|
||||
— `deploy.py` runs with it as the working directory, so `files.template('nginx.conf.j2', ...)`
|
||||
resolves.
|
||||
|
||||
This is the one constraint that does not exist while the cubes live in your own
|
||||
repo, which makes it the one most likely to be discovered late. Test against an
|
||||
installed copy, not a linked one.
|
||||
|
||||
## How resolution actually works
|
||||
|
||||
Worth knowing, because two of the failure modes are otherwise baffling.
|
||||
|
||||
**Where a package is looked up from.** Each `cubePackages` entry is resolved from
|
||||
the directory of the config file that named it, not from the working directory.
|
||||
Configs merge upward, so a `.nopyrc.json` two levels up can name a bundle that
|
||||
only exists in *its* `node_modules`, and it resolves. The lookup reads
|
||||
`package.json` off disk via `createRequire(...).resolve.paths()` rather than
|
||||
going through `exports` — a bundle ships directories and has no entry point to
|
||||
declare.
|
||||
|
||||
**Why the loader does not simply scan `node_modules`.** It cannot: pnpm plants a
|
||||
symlink at `node_modules/<name>`, and `readdir` reports it as a symlink, not a
|
||||
directory, so a recursive scan skips every package silently. Naming packages
|
||||
explicitly is the fix, and it is also the reason `node_modules` is skipped during
|
||||
the cube scan itself.
|
||||
|
||||
**How a manifest finds its imports.** Ordinary Node resolution, from the
|
||||
manifest's own directory. An installed bundle has its own `node_modules` with
|
||||
`@bitsquare/nopy-cube` and `zod` in it, so this just works. A hand-written cube
|
||||
sitting in a directory with no `node_modules` would historically fail with
|
||||
`ERR_MODULE_NOT_FOUND`; nopy now registers a resolve hook that catches exactly
|
||||
that case and falls back to resolving `@bitsquare/nopy-cube`, `@bitsquare/nopy`
|
||||
and `zod` from the running CLI. Normal resolution is always tried first, so a
|
||||
cube that ships its own zod keeps it. Treat the hook as a convenience for local
|
||||
cubes — a published bundle must declare its dependencies properly.
|
||||
|
||||
**Two copies of zod is a real hazard.** `instanceof` comparisons fail across
|
||||
copies, which is why nopy inspects schemas structurally (`schema.def.type`) and
|
||||
why `secrets` is a plain array rather than `.meta()` metadata — zod's metadata
|
||||
registry is per-copy, and a marker written into one copy's registry is invisible
|
||||
to another. Keep your zod range compatible with the CLI's (`^4.4.3`) and the
|
||||
package manager will usually give you one copy.
|
||||
|
||||
## Publishing
|
||||
|
||||
Nothing bundle-specific: `npm publish` (or `pnpm publish`) with a version bump.
|
||||
Some things worth deciding once:
|
||||
|
||||
- **Version the bundle independently of nopy.** There is no compatibility check
|
||||
between the two — the loader reads whatever `nopy.cubes` points at. Document
|
||||
the nopy version you test against in your README.
|
||||
- **Renaming or removing an id is breaking.** It invalidates recorded sessions
|
||||
and breaks any manifest listing it as a dependency, including manifests in
|
||||
other people's bundles.
|
||||
- **Changing a schema key is breaking** in the same way; adding one with a
|
||||
`.default()` is not.
|
||||
- **Test the installed shape, not the linked one.** `npm pack`, install the
|
||||
tarball into a throwaway directory with a `.nopyrc.json` naming it, and deploy
|
||||
from it. This is what catches a missing file, a cube that writes to its own
|
||||
directory, and an undeclared dependency — none of which show up while the
|
||||
package is symlinked into the repo that wrote it.
|
||||
|
||||
For an unattended check, replay a session file rather than reaching for `-P`
|
||||
alone, which still opens the interactive picker:
|
||||
|
||||
```sh
|
||||
nopy install -l session.json -P -D
|
||||
```
|
||||
|
||||
Note that a replay re-prompts for anything a manifest lists in `secrets` —
|
||||
those are never written to a session — so pick a cube without them, or put the
|
||||
values under `env` in `.nopyrc.json`.
|
||||
|
||||
For how this repository releases its own packages, see
|
||||
[README.PUBLISH.md](../../../README.PUBLISH.md).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause |
|
||||
| --- | --- |
|
||||
| `Cube package 'X' is not installed (looked up from …)` | Not installed, or installed somewhere other than the config that named it. The path in the message is where the lookup started. |
|
||||
| `Cube package 'X' declares no cubes` | Missing or malformed `nopy.cubes` in the package's `package.json`. It must be a non-empty array of strings. |
|
||||
| `'./cubes' does not exist in …` | The directory was not packed. Check `files` and `npm pack --dry-run`. |
|
||||
| `'…' points outside the package` | A `nopy.cubes` entry escaping the package root. Not allowed. |
|
||||
| `Duplicate cube id 'X' from N sources:` | Two or more cubes claiming one id; the message lists each source. Rename one — there is no precedence rule to lean on. |
|
||||
| `ERR_MODULE_NOT_FOUND` for `zod` or `@bitsquare/nopy-cube` | The bundle did not declare them as dependencies. The resolve-hook fallback covers loose local cubes, not published packages. |
|
||||
| `Invalid manifest in …: 'secrets' names X, which is not in the schema` | A `secrets` entry with no matching schema key — usually a typo or a renamed field. |
|
||||
| Cubes work linked, fail installed | Almost always a write into the cube's own directory, or a file missing from `files`. |
|
||||
@@ -0,0 +1,677 @@
|
||||
# Cube bundles as npm packages
|
||||
|
||||
Status: **All six phases have landed. This document is now a record, not a plan.**
|
||||
The one thing still unproven is the publish lane against a real registry — see
|
||||
*Risks*.
|
||||
`cubePackages` resolves and loads end to end, `@bitsquare/nopy-cube` exists and
|
||||
the publish lane can ship a linked package. What is missing is a bundle to point
|
||||
`cubePackages` at.
|
||||
|
||||
Distributing cubes as npm packages so a project can `pnpm add @acme/cubes-net`
|
||||
and have its cubes show up in `nopy` alongside local ones.
|
||||
|
||||
## Goals
|
||||
|
||||
- A cube bundle is an ordinary npm package, publishable to npmjs or Gitea
|
||||
through the existing release lanes.
|
||||
- A consuming project opts into a bundle explicitly, by name, in `.nopyrc.json`.
|
||||
- Existing manifests, dependency specs (`dependencies: () => ['apt:essentials']`)
|
||||
and stored session history keep working untouched.
|
||||
- The in-repo `cubes/` tree becomes the first published bundle, proving the path.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Automatic discovery of bundles from the dependency tree. Cubes run privileged
|
||||
deploy scripts against real hosts; a transitive dependency contributing one
|
||||
silently is a supply-chain hole. Opt-in per package, always.
|
||||
- Namespacing or id rewriting. Ids stay flat and global (see *Decisions*).
|
||||
- Version compatibility checks between a bundle and the `nopy` running it.
|
||||
Noted as a risk, deferred.
|
||||
|
||||
## Decisions
|
||||
|
||||
| Question | Decision |
|
||||
| --- | --- |
|
||||
| Duplicate cube ids across sources | **Hard error.** No precedence, no shadowing. Mitigation is a good error message, not a fallback. |
|
||||
| Id format | Unchanged, flat. The id is the session key (`dependencies.ts:135`); changing it breaks `--repeat-last` and `--history`. |
|
||||
| Discovery | Explicit `cubePackages` list in `.nopyrc.json`. |
|
||||
| Migrate in-repo `cubes/` | Yes — `packages/cubes-core`, as the proof of concept. |
|
||||
| Split an authoring package (`@bitsquare/nopy-cube`) | **Yes.** Bundles take a regular dependency on it; `@bitsquare/nopy` re-exports it for backwards compatibility. See *Phase 4*. |
|
||||
|
||||
## Current state
|
||||
|
||||
What already works, unchanged:
|
||||
|
||||
- `--chdir <cubeDir>` (`nopy.executor.ts`) means a `deploy.py` under
|
||||
`node_modules` runs fine; pyinfra only needs the path.
|
||||
- `scanDirectory` skips `node_modules` when *descending* (`loader.ts:101`), not
|
||||
for the root it is handed. So `"cubeDirs": ["./node_modules/@acme/cubes-net/cubes"]`
|
||||
works today. That is the escape hatch until this lands, and it stays working
|
||||
afterwards.
|
||||
|
||||
What blocks a clean story:
|
||||
|
||||
1. **Manifest imports.** `manifest.mjs` does `import { cubes } from '@bitsquare/nopy'`,
|
||||
resolved by ordinary Node resolution from the manifest's own directory. From
|
||||
inside `node_modules/@acme/cubes-net/`, that resolves upward into the
|
||||
consumer's `node_modules` — fine if the consumer installed `@bitsquare/nopy`,
|
||||
`ERR_MODULE_NOT_FOUND` if `nopy` is only installed globally. Same gotcha
|
||||
CLAUDE.md already documents for the local `cubes/` tree.
|
||||
2. **No way to name a package** in config, only paths.
|
||||
3. **Recursively scanning `node_modules` is not a workaround.** pnpm symlinks
|
||||
direct deps, and `readdir(withFileTypes)` reports a symlink as
|
||||
`isSymbolicLink()`, not `isDirectory()` — the scan would skip every package.
|
||||
Package roots must be resolved explicitly.
|
||||
|
||||
## Phase 0 — fixes that land first — **done**
|
||||
|
||||
Independent of packaging, and the duplicate-id work depends on them.
|
||||
|
||||
**0.1 `scanDirectory` drops subtrees on duplicates.** `loader.ts:84-87` pushed
|
||||
the error and `return`ed, which exited before the recursive descent at line 100.
|
||||
Cubes nested below a duplicate never got scanned, so the error report was
|
||||
incomplete: you fix one collision, re-run, find the next.
|
||||
|
||||
**0.2 Duplicate detection is order-dependent.** `loadCubes()` ran `Promise.all`
|
||||
over folders into a shared `cubes` object, so which source was "first" and which
|
||||
was "the duplicate" varied run to run.
|
||||
|
||||
Both are one restructure. Scanning and id resolution are now separate passes:
|
||||
each root fills its own `ScanResult`, the lists are concatenated in root order
|
||||
(`Promise.all` preserves input order regardless of completion order), and a
|
||||
grouping pass builds `cubes` and the errors. `scanDirectory` no longer decides
|
||||
anything about ids, so it always descends. Directory entries are sorted, and a
|
||||
directory reachable from two roots is deduped by path — one cube seen twice is
|
||||
not a collision, which it used to be reported as.
|
||||
|
||||
**0.3 `apt:essentials` is already declared twice.** `cubes/apt/essentials`
|
||||
declares it via `id`; `packages/nopy/cubes/apt/essentials` declared it via an
|
||||
`[apt:essentials]` prefix in `name`. `cubeDirs` merges root-first, so running
|
||||
`nopy` from `packages/nopy` collected both and aborted. Confirmed against the
|
||||
real trees before the rename:
|
||||
|
||||
```
|
||||
Duplicate cube id 'apt:essentials' from 2 sources:
|
||||
/…/ansiblingz/cubes/apt/essentials
|
||||
/…/ansiblingz/packages/nopy/cubes/apt/essentials
|
||||
Rename one of them, or remove a source from .nopyrc.json.
|
||||
```
|
||||
|
||||
The three `packages/nopy/cubes` fixtures are now `[test:apt-essentials]`,
|
||||
`[test:apt-all]` and `[test:apt-more]`; all 25 cubes load with no errors. Their
|
||||
`dependencies` were stale too — they named `apt/more` and `apt/essentials`,
|
||||
which are not ids anything declares — so they now point at the renamed ids.
|
||||
|
||||
**0.4 `coerceValue` breaks if zod is ever duplicated.** `nopy.prompts.ts:147-154`
|
||||
discriminated with `instanceof z.ZodDefault`, `z.ZodBoolean`, `z.ZodNumber` and
|
||||
friends — checks against the *running CLI's* zod instance. The moment a bundle
|
||||
resolves its own copy of zod (entirely possible once manifests arrive from
|
||||
`node_modules`; see Phase 4), every check returns false and `coerceValue` falls
|
||||
through to the raw string, silently. Booleans stop being booleans.
|
||||
|
||||
`defaultValueOf` in `cubes/types.ts` had the same breakage, reached whenever a
|
||||
schema has one field without a `.default()` — `getDefaults()` tries
|
||||
`safeParse({})` first, which is instance-agnostic, and only then drops to the
|
||||
per-field read.
|
||||
|
||||
Both now discriminate on `def.type`, a plain string that holds across instances,
|
||||
via two exported helpers (`zodKind`, `zodInner`). Verified on the installed zod
|
||||
4.4.3:
|
||||
|
||||
```
|
||||
z.boolean().default(false).def.type → 'default'
|
||||
z.boolean().default(false).def.innerType → { def: { type: 'boolean' } }
|
||||
z.number().def.type → 'number'
|
||||
```
|
||||
|
||||
`tests/helpers/foreign-zod.ts` rebuilds a schema as plain objects carrying zod's
|
||||
`def` but not its prototype — structurally what a second copy of zod produces,
|
||||
and `instanceof`-blind, so neither call site can regress.
|
||||
|
||||
Worth noting for Phase 4: zod 4 exposes `def.defaultValue` as a getter that
|
||||
already invokes a lazily declared default, so the `typeof === 'function'` branch
|
||||
in `defaultValueOf` is now dead. It is kept as insurance against that changing.
|
||||
|
||||
## Phase 1 — the bundle contract
|
||||
|
||||
A cube bundle is an npm package with a `nopy` field:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@acme/cubes-net",
|
||||
"version": "1.0.0",
|
||||
"type": "module",
|
||||
"nopy": { "cubes": ["./cubes"] },
|
||||
"files": ["cubes", "README.md", "LICENSE"],
|
||||
"keywords": ["nopy", "nopy-cubes", "pyinfra"],
|
||||
"dependencies": {
|
||||
"@bitsquare/nopy-cube": "^1.0.0",
|
||||
"zod": "^4.4.3"
|
||||
},
|
||||
"publishConfig": { "access": "public" }
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `nopy.cubes` — directories relative to the package root, scanned exactly like
|
||||
`cubeDirs` entries. Required; a package listed in `cubePackages` without a
|
||||
`nopy` field is an error, not a silent skip. Listing it means the user expects
|
||||
cubes from it.
|
||||
- Both dependencies are **regular dependencies, not peers**, and both are
|
||||
load-bearing: a manifest imports `Manifest` from `@bitsquare/nopy-cube` and `z`
|
||||
from `zod`. `@bitsquare/nopy-cube` peer-depends on zod, so the bundle's copy is
|
||||
the one everybody uses — see Phase 4.
|
||||
- The package needs no `exports` entry for this to work — resolution reads
|
||||
`package.json` off disk (Phase 2), so the `exports` map is irrelevant.
|
||||
- **A bundle's directory is read-only at runtime.** Under pnpm, `node_modules`
|
||||
content is hardlinked into the global store; a cube writing next to its own
|
||||
`deploy.py` corrupts that store for every project on the machine. Cubes must
|
||||
write to `/tmp` or the remote host, never their own dir.
|
||||
- A bundle must not ship a `.nopyrc.json`. Config discovery walks up from
|
||||
`process.cwd()`, never from cube directories, so it would never be read.
|
||||
|
||||
## Phase 2 — resolution — **done**
|
||||
|
||||
### Config surface
|
||||
|
||||
```json
|
||||
{
|
||||
"cubePackages": ["@acme/cubes-net", "@acme/cubes-caddy"]
|
||||
}
|
||||
```
|
||||
|
||||
Merges through the existing `resolution` machinery for free — arrays concat and
|
||||
dedupe — so a parent config supplies the org baseline and a child adds to it.
|
||||
|
||||
**Resolution origin.** A package must be resolved from *the directory of the
|
||||
config file that declared it*, not from `process.cwd()`. Otherwise a bundle
|
||||
listed in `~/.nopyrc.json` cannot resolve unless every project happens to depend
|
||||
on it. This is the same problem `PATH_PROPERTIES` solves for `cubeDirs`, but the
|
||||
output is a tagged reference rather than a rewritten string:
|
||||
|
||||
```ts
|
||||
export interface CubePackageRef {
|
||||
spec: string; // '@acme/cubes-net'
|
||||
from: string; // dirname of the .nopyrc.json that declared it
|
||||
}
|
||||
```
|
||||
|
||||
So the file format and the loaded format diverge for this one key:
|
||||
|
||||
```ts
|
||||
interface NopyConfigFile extends Omit<Partial<NopyConfig>, 'cubePackages'> {
|
||||
cubePackages?: string[];
|
||||
resolution?: ResolutionConfig;
|
||||
}
|
||||
|
||||
interface NopyConfig {
|
||||
cubePackages: CubePackageRef[];
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
`resolveConfigPaths()` performs the `string → CubePackageRef` conversion, next to
|
||||
where it resolves `PATH_PROPERTIES`. Two consequences to handle:
|
||||
|
||||
- `mergeValue`'s array dedupe only fires when every element is a primitive
|
||||
(`config.ts:142`), so refs fall through to plain concat. Dedupe by `spec` in
|
||||
the resolver instead.
|
||||
- Dedupe is **last-wins**: merge order is root-first, so the last occurrence is
|
||||
the most specific config, and its `from` is the right resolution origin.
|
||||
|
||||
### Resolver
|
||||
|
||||
New file `packages/nopy/src/cubes/packages.ts`:
|
||||
|
||||
```ts
|
||||
export interface CubePackage {
|
||||
name: string;
|
||||
root: string;
|
||||
dirs: string[]; // absolute, from nopy.cubes
|
||||
}
|
||||
|
||||
export function resolveCubePackages(
|
||||
refs: CubePackageRef[]
|
||||
): { packages: CubePackage[]; errors: string[] };
|
||||
```
|
||||
|
||||
Locate the package root without going through `exports` and without tripping on
|
||||
pnpm symlinks:
|
||||
|
||||
```ts
|
||||
const req = createRequire(path.join(ref.from, 'noop.js'));
|
||||
for (const dir of req.resolve.paths(ref.spec) ?? []) {
|
||||
const manifest = path.join(dir, ref.spec, 'package.json');
|
||||
if (fs.existsSync(manifest)) return path.dirname(manifest);
|
||||
}
|
||||
```
|
||||
|
||||
`resolve.paths()` walks the `node_modules` chain upward from `ref.from` plus the
|
||||
global paths; `existsSync` follows symlinks, so pnpm's
|
||||
`node_modules/@acme/cubes-net → ../.pnpm/…` resolves correctly.
|
||||
|
||||
Errors (each aborts the run, consistent with the existing `errors` contract):
|
||||
|
||||
- package not found on any candidate path
|
||||
- `package.json` unparseable
|
||||
- no `nopy.cubes`, or it is not a non-empty array of strings
|
||||
- a `nopy.cubes` entry escapes the package root, or does not exist
|
||||
|
||||
### Wiring
|
||||
|
||||
`findCubeDirectories()` currently returns `string[]`. It becomes the union of
|
||||
three sources, each tagged so the loader can attribute a cube to it:
|
||||
|
||||
```ts
|
||||
export type CubeRoot =
|
||||
| { type: 'dir'; dir: string } // cubeDirs, .npcubes markers
|
||||
| { type: 'package'; dir: string; packageName: string }; // cubePackages
|
||||
|
||||
export function findCubeRoots(): { roots: CubeRoot[]; errors: string[] };
|
||||
```
|
||||
|
||||
Keep `findCubeDirectories()` as a thin wrapper returning `roots.map(r => r.dir)` —
|
||||
it is exported from `src/cubes/index.ts` and covered by tests. The
|
||||
`node_modules` skip inside `scanDirectory` stays and is now *correct*: a
|
||||
bundle's own `node_modules` should not be scanned.
|
||||
|
||||
## Phase 3 — hard errors with attribution — **done**
|
||||
|
||||
`Cube` gains a source, as an optional fourth constructor parameter so the public
|
||||
signature stays backwards compatible:
|
||||
|
||||
```ts
|
||||
export type CubeSource =
|
||||
| { type: 'dir'; dir: string }
|
||||
| { type: 'package'; packageName: string; dir: string };
|
||||
|
||||
class Cube {
|
||||
constructor(
|
||||
manifest: Manifest<Schema>,
|
||||
dir: string,
|
||||
deployScript: string,
|
||||
source: CubeSource = { type: 'dir', dir }
|
||||
) {}
|
||||
}
|
||||
```
|
||||
|
||||
The duplicate error carries both sources and is order-independent (Phase 0.2):
|
||||
|
||||
```
|
||||
Duplicate cube id 'apt:essentials' from 2 sources:
|
||||
package @bitsquare/cubes-core /…/node_modules/@bitsquare/cubes-core/cubes/apt/essentials
|
||||
directory /repo/packages/nopy/cubes/apt/essentials
|
||||
Rename one of them, or remove a source from .nopyrc.json.
|
||||
```
|
||||
|
||||
There is deliberately no override, alias or precedence rule. If two bundles ever
|
||||
claim the same id they are mutually exclusive, and the fix is upstream.
|
||||
|
||||
Surface the source in the interactive picker and in `--json` output so a user can
|
||||
see where a cube came from before running it.
|
||||
|
||||
## Phase 4 — `@bitsquare/nopy-cube`, the authoring package — **done**
|
||||
|
||||
The problem: a manifest does `import { cubes } from '@bitsquare/nopy'`, resolved
|
||||
by ordinary Node resolution from the manifest's own directory. From inside
|
||||
`node_modules/@acme/cubes-net/`, that only resolves if the consumer installed
|
||||
`@bitsquare/nopy` locally — a globally-installed CLI leaves nothing to find.
|
||||
|
||||
The fix is to give bundles something they can depend on *normally*, so resolution
|
||||
is plain, boring, spec-compliant Node with no loader tricks in the critical path.
|
||||
|
||||
### The package
|
||||
|
||||
`packages/nopy-cube` — the `Manifest` factory, the `Cube` class, and the types
|
||||
from `cubes/types.ts`. No CLI, no `execa`, `inquirer`, `enquirer`, `zx`, or
|
||||
`commander`. Today a cube manifest — a file that ships nothing but data — drags
|
||||
the entire CLI in as a transitive dependency; this makes the authoring surface
|
||||
honest about how small it is, and gives the *contract* a version number that
|
||||
moves independently of the CLI's.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@bitsquare/nopy-cube",
|
||||
"version": "1.0.0-alpha0",
|
||||
"type": "module",
|
||||
"exports": { ".": { "types": "./dist/index.d.ts", "default": "./dist/index.js" } },
|
||||
"peerDependencies": { "zod": "^4.4.3" },
|
||||
"files": ["dist", "README.md", "LICENSE"]
|
||||
}
|
||||
```
|
||||
|
||||
**zod is a peer, deliberately.** Bundles declare zod as a regular dependency, so
|
||||
exactly one zod instance serves the manifest, the schema it builds, and the
|
||||
`Manifest` factory. Phase 0.4 removes the CLI's `instanceof` dependence on that
|
||||
being the *same* copy the CLI uses, but keeping the bundle side single-instance
|
||||
is still the right default.
|
||||
|
||||
### Moving `cubes/types.ts`
|
||||
|
||||
`@bitsquare/nopy` re-exports everything from `@bitsquare/nopy-cube` — through
|
||||
`src/cubes/index.ts` and the `cubes` namespace in `src/nopy.cubes.ts`, both
|
||||
already coverage-excluded barrels — so `import { cubes } from '@bitsquare/nopy'`
|
||||
in every existing manifest keeps working unchanged. Nothing in `cubes/` has to be
|
||||
touched at migration time.
|
||||
|
||||
`cubes/types.ts` and `cubes/factories.ts` moved wholesale, with
|
||||
`tests/cubes.types.test.ts` and `tests/cubes.factories.test.ts` behind them.
|
||||
`tests/helpers/foreign-zod.ts` is duplicated rather than shared — fifteen lines,
|
||||
and the alternative is a test-only dependency edge between the packages.
|
||||
|
||||
Repo plumbing it took:
|
||||
|
||||
- `tsconfig.base.json`: `"@bitsquare/nopy-cube": ["./packages/nopy-cube/src"]`.
|
||||
- Root `tsconfig.json` and `packages/nopy/tsconfig.json`: the project reference.
|
||||
This is the first reference edge in the repo, and it broke the gate
|
||||
immediately: **`tsc --build --noEmit` is not legal once a project has
|
||||
references** — TS6310, "referenced project may not disable emit", because a
|
||||
composite project has to emit the declarations its dependents read. The root
|
||||
`typecheck` script is now plain `tsc --build`. It still fails on a type error,
|
||||
and it now also proves the build works; the cost is that it writes `dist`,
|
||||
which is gitignored.
|
||||
- `packages/nopy/package.json`: `"@bitsquare/nopy-cube": "workspace:*"`.
|
||||
- `packages/nopy/vitest.config.ts`: a `resolve.alias` for `@bitsquare/nopy-cube`
|
||||
pointing at `../nopy-cube/src/index.ts`. Without it the workspace link
|
||||
resolves through `exports` to `dist`, so `pnpm test` on a clean checkout would
|
||||
fail until something had built it, and a stale `dist` would silently be what
|
||||
the tests ran against. The same config excludes `**/nopy-cube/**` from
|
||||
coverage — the aliased files were being counted against nopy's thresholds.
|
||||
- A `vitest.config.ts` for the new package with the same thresholds. It sits at
|
||||
100 % statements/functions/lines, 91 % branches.
|
||||
|
||||
### The release lane needed fixing first
|
||||
|
||||
This is the part that is easy to miss. `link-workspace-packages` is unset and
|
||||
pnpm 10+ defaults it to `false`, so a plain semver range would resolve
|
||||
`@bitsquare/nopy-cube` from the registry instead of linking the workspace copy —
|
||||
the dependency has to use `workspace:*`.
|
||||
|
||||
But **both workflows publish with `npm publish`**, and npm does not understand
|
||||
the `workspace:` protocol. `@bitsquare/nopy` would ship a manifest carrying
|
||||
`"@bitsquare/nopy-cube": "workspace:*"`, which fails on install with
|
||||
`EUNSUPPORTEDPROTOCOL`. This has never mattered because the two current packages
|
||||
do not depend on each other; `nopy → nopy-cube` is the first edge, and the PoC
|
||||
bundle in Phase 5 adds a second.
|
||||
|
||||
Pick one before publishing anything:
|
||||
|
||||
- **Switch to `pnpm publish --no-git-checks`**, which rewrites the protocol to a
|
||||
concrete version on pack. Cleanest, but changes the publish step in both
|
||||
workflows and pulls in pnpm's own lifecycle behaviour.
|
||||
- **Rewrite the range with `npm pkg set` before publishing**, extending the
|
||||
pattern `publish-snapshot.yml` already uses for `version`. In `release.yml` one
|
||||
package ships at a time, so it pins to whatever version `packages/nopy-cube/package.json`
|
||||
declares at that commit. In `publish-snapshot.yml` the loop needs to become two
|
||||
passes — compute every snapshot version first, then publish — so `nopy` can pin
|
||||
the exact `nopy-cube` snapshot from the same run.
|
||||
|
||||
**Measured, both directions.** `npm pack` in `packages/nopy` produces a tarball
|
||||
whose manifest still reads `"@bitsquare/nopy-cube": "workspace:*"`; `pnpm pack`
|
||||
produces one that reads `"1.0.0-alpha0"`. So the failure was real and the fix
|
||||
works.
|
||||
|
||||
Went with `pnpm publish --ignore-scripts --no-git-checks` in both workflows.
|
||||
`--no-git-checks` is not optional in either: `release.yml` runs on a detached
|
||||
HEAD, and `publish-snapshot.yml` dirties the tree by stamping versions.
|
||||
(`pnpm pack` has no `--ignore-scripts`, only `pnpm publish` does.)
|
||||
|
||||
Three small scripts carry the parts that are easy to get wrong, all runnable
|
||||
locally:
|
||||
|
||||
- **`scripts/verify-pack.mjs`** — packs every publishable package and fails if a
|
||||
`workspace:` range survived into the tarball. Runs between build and publish
|
||||
in both workflows. Turns "npm would have shipped a broken manifest" from an
|
||||
install-time surprise into a red run.
|
||||
- **`scripts/publish-order.mjs`** — topologically sorts the publishable packages.
|
||||
`packages/*/` alphabetically puts `nopy` ahead of the `nopy-cube` it depends
|
||||
on; the snapshot workflow now iterates this instead.
|
||||
- **`scripts/linked-deps.mjs`** — lists a package's workspace links as
|
||||
`<name> <version>`, resolved by package name rather than by directory.
|
||||
`release.yml` uses it to refuse a release whose linked dependency is not on
|
||||
npmjs yet, which is the one mistake that cannot be taken back after 72 hours.
|
||||
|
||||
`publish-snapshot.yml` also became two passes over the packages: stamp every
|
||||
version first, then publish. `pnpm publish` substitutes the version the linked
|
||||
package declares *at pack time*, so `nopy-cube` has to be carrying its snapshot
|
||||
version before `nopy` is packed.
|
||||
|
||||
Still unverified: none of this has run against the Gitea registry. Worth a
|
||||
throwaway version before the first real release.
|
||||
|
||||
### Also: the resolve hook — built
|
||||
|
||||
Independent of the split, and worth building anyway — it retires the
|
||||
`ERR_MODULE_NOT_FOUND` gotcha CLAUDE.md documents for the local `cubes/` tree,
|
||||
where manifests import `@bitsquare/nopy` from a directory that has no link to it.
|
||||
|
||||
With the split, the hook is a convenience rather than load-bearing: bundles
|
||||
resolve `@bitsquare/nopy-cube` through their own `node_modules` and never reach
|
||||
it.
|
||||
|
||||
**The gotcha is bigger than CLAUDE.md says: it is two specifiers, not one.**
|
||||
Measured by linking `@bitsquare/nopy` into the root `node_modules` and loading
|
||||
the real tree — every manifest then failed on `Cannot find package 'zod'`
|
||||
instead. Manifests import `z` directly to build their schema, and pnpm's
|
||||
isolated layout puts zod under `packages/nopy/node_modules`, not the root. A
|
||||
hook that only covers `@bitsquare/nopy` moves the error rather than fixing it,
|
||||
so it has to fall back for `zod` too. With both linked, all 25 cubes load.
|
||||
|
||||
Falling back for `zod` hands local cubes the *CLI's* zod instance, so no
|
||||
duplication arises there. Bundles are the case that duplicates it, and Phase 0.4
|
||||
is what makes that safe.
|
||||
|
||||
`packages/nopy/src/cubes/resolve-hook.mjs`, registered once from `loadCubes()`
|
||||
before the first `import(manifestPath)`:
|
||||
|
||||
```ts
|
||||
module.register('./resolve-hook.mjs', import.meta.url, { data: { from: import.meta.url } });
|
||||
```
|
||||
|
||||
`from` is a URL inside the running CLI's own package; the hook thread builds a
|
||||
`createRequire` from it and resolves the fallbacks out of the CLI's own
|
||||
dependencies.
|
||||
|
||||
The hook tries `next(specifier, ctx)` **first** and only falls back on failure.
|
||||
That ordering matters: a consumer that has its own copy installed keeps using
|
||||
it, so the hook never silently introduces version skew. There is a test for
|
||||
exactly that — a stub `zod` beside the cube wins over the CLI's real one.
|
||||
|
||||
Constraints, as built:
|
||||
|
||||
- `module.register()` is process-global and cannot be undone. Installed once,
|
||||
behind a module-level guard, and wrapped in a `try` — the hook is a
|
||||
convenience, so a registration failure must not abort a run.
|
||||
- The hook file runs on a separate thread; the `data` payload must be
|
||||
structured-cloneable (a string URL is).
|
||||
- The `.mjs` has to reach `dist`, and `tsc` does not copy it: nopy's `build` is
|
||||
now `tsc && cp src/cubes/*.mjs dist/cubes/`. `files` already covers it via the
|
||||
`dist` entry.
|
||||
- It covers **three** specifiers, not the two the plan named: `zod`,
|
||||
`@bitsquare/nopy`, and `@bitsquare/nopy-cube` — a hand-written local cube is
|
||||
as entitled to the new authoring package as to the old one. Subpaths count
|
||||
(`@bitsquare/nopy/package.json`), anything else stays a hard failure.
|
||||
|
||||
**The tests have to spawn a real `node`.** Written inside the vitest worker they
|
||||
pass whether or not the hook is installed: vite resolves the dynamic import
|
||||
itself and finds `zod` from the project root. `tests/cubes.resolve-hook.test.ts`
|
||||
therefore runs each case in a child process, and the first case asserts the
|
||||
*failure* without the hook so the rest cannot silently stop proving anything.
|
||||
|
||||
**Verified end to end.** From a plain `node` at the repo root, with nothing
|
||||
linked, the built loader reads all 22 cubes under `cubes/` with zero errors. The
|
||||
`ERR_MODULE_NOT_FOUND` gotcha in `CLAUDE.md` is retired.
|
||||
|
||||
## Phase 5 — proof of concept: `packages/cubes-core` — **done**
|
||||
|
||||
Depends on Phase 4 shipping first — the bundle cannot declare
|
||||
`@bitsquare/nopy-cube` as a dependency until it exists, and the publish-lane fix
|
||||
has to be in place before either package is published.
|
||||
|
||||
1. `git mv cubes packages/cubes-core/cubes` — preserves per-file history.
|
||||
2. Add `packages/cubes-core/package.json` per the Phase 1 contract. Version
|
||||
`1.0.0-alpha0`, tracking the current alpha train. Not private. Its
|
||||
`@bitsquare/nopy-cube` dependency uses `workspace:*` in the repo, which is
|
||||
exactly the case the Phase 4 publish fix has to handle.
|
||||
Migrating the manifests' `import { cubes } from '@bitsquare/nopy'` to
|
||||
`import { Manifest } from '@bitsquare/nopy-cube'` is optional — the re-export
|
||||
keeps the old form working — but doing it here is what proves the bundle
|
||||
resolves without the CLI present at all.
|
||||
3. Root `.nopyrc.json`: **replace** `"cubeDirs": ["./cubes"]` with
|
||||
`"cubePackages": ["@bitsquare/cubes-core"]`. Replace, not add — keeping both
|
||||
means every id resolves from two sources and the hard error fires on every
|
||||
run.
|
||||
4. Root `package.json`: add `"@bitsquare/cubes-core": "workspace:*"` to
|
||||
`devDependencies`, so pnpm symlinks it into the root `node_modules`. This is
|
||||
what makes the PoC exercise the real pnpm symlink resolution path rather than
|
||||
a plain directory.
|
||||
5. `packages/nopy/.nopyrc.json` keeps `"cubeDirs": ["./cubes"]` for its fixtures.
|
||||
Config merges root-first, so running from `packages/nopy` now pulls in
|
||||
`@bitsquare/cubes-core` *and* the fixtures — which is exactly the collision
|
||||
Phase 0.3 renames away.
|
||||
6. Workflow changes are limited to the publish-lane fix from Phase 4.
|
||||
`publish-snapshot.yml` loops `for dir in packages/*/` and picks both new
|
||||
packages up automatically; `release.yml` resolves `packages/<pkg>` from the
|
||||
tag, so `cubes-core-v1.0.0` and `nopy-cube-v1.0.0` work as-is. Verify on the
|
||||
first snapshot run that a package with no `build` script is skipped cleanly by
|
||||
`pnpm -r run build` (it is) and that publishing is happy with no lifecycle
|
||||
scripts.
|
||||
7. No `tsconfig` reference for `cubes-core` — the bundle has no TypeScript. (The
|
||||
`nopy-cube` references from Phase 4 are separate.)
|
||||
8. Biome already lints `cubes/**/*.mjs` from the root; only the path changes.
|
||||
|
||||
### What differed from the plan
|
||||
|
||||
- **Step 2's optional migration was done.** All 22 manifests now import
|
||||
`{ Manifest }` from `@bitsquare/nopy-cube`, not `{ cubes }` from
|
||||
`@bitsquare/nopy`. Optional for correctness, but it is the only version of the
|
||||
PoC that proves anything: leaving the old import in place would have resolved
|
||||
through the CLI that happens to sit in the same tree.
|
||||
- **`uniqid` had to move too.** Two manifests use it (`admin:hostname` bare,
|
||||
`user:add` via `cubes.uniqid`), so `src/cubes/utils.ts` and its test went to
|
||||
`nopy-cube` alongside `types.ts`, and `uniqid` joined the authoring barrel.
|
||||
Otherwise one migrated manifest would still have been importing the CLI.
|
||||
- **`files` needs a log exclusion.** Cubes that have been run leave a gitignored
|
||||
`pyinfra-debug.log` next to `deploy.py`; gitignore does not filter an npm
|
||||
tarball. `"files": ["cubes", "!cubes/**/*.log", …]` does. Verified: 22
|
||||
manifests, 22 deploy scripts, 0 logs in the packed artefact.
|
||||
- **`verify-pack.mjs` picks the bundle up for free** — it walks every non-private
|
||||
`packages/*`, so `cubes-core`'s `workspace:*` edge is checked like nopy's.
|
||||
|
||||
### Verifying the PoC — done
|
||||
|
||||
- **In-workspace:** the built loader, run from the repo root against the new
|
||||
root `.nopyrc.json`, reads 22 cubes with 0 errors and reports
|
||||
`source: { type: 'package', packageName: '@bitsquare/cubes-core', dir:
|
||||
'…/node_modules/@bitsquare/cubes-core/cubes' }` — the pnpm symlink path, not a
|
||||
plain directory.
|
||||
- **Out-of-workspace (the real test):** `pnpm pack` for `nopy-cube`, `nopy` and
|
||||
`cubes-core`, then **`npm install`** of all three tarballs into a throwaway
|
||||
directory with a `.nopyrc.json` naming only the bundle. npm is the strict test
|
||||
here — it does not understand `workspace:`, so a leaked range fails the install
|
||||
outright. It installed clean, and the installed
|
||||
`@bitsquare/nopy/package.json` carries `"@bitsquare/nopy-cube":
|
||||
"1.0.0-alpha0"`. `nopy install -l session.json -P -D` then resolved
|
||||
`apt:essentials` and printed a `--chdir` into
|
||||
`node_modules/@bitsquare/cubes-core/cubes/apt/essentials`. Since the loader
|
||||
aborts on any manifest error and this run did not, all 22 manifests imported
|
||||
`@bitsquare/nopy-cube` and `zod` successfully from a tree containing no
|
||||
workspace links.
|
||||
|
||||
Note for anyone repeating this: `-P` on its own is interactive, and a replay
|
||||
still prompts for anything a manifest declares in `secrets` (they are never
|
||||
persisted to a session) — `net:tailscale` will sit there waiting. Use a
|
||||
session file with a cube that has no secrets, or answer the prompt.
|
||||
|
||||
## Phase 6 — documentation — **done**
|
||||
|
||||
- `CLAUDE.md`: the repo table gains two rows (`packages/nopy-cube`,
|
||||
`packages/cubes-core`) and loses the `cubes/` one; "The two packages do not
|
||||
depend on each other" is no longer true; the loader section in *nopy
|
||||
architecture*; and the *Gotcha* paragraph, which the resolve hook retires.
|
||||
- `packages/nopy/docs/CUBE-BUNDLES.md` (new): authoring guide — package shape,
|
||||
read-only constraint, id collision policy, publishing.
|
||||
- `packages/nopy/docs/API.md` + `README.md`: `cubePackages`.
|
||||
- `README.PUBLISH.md`: `nopy-cube-v*` and `cubes-core-v*` as new tag prefixes,
|
||||
plus the ordering constraint — `nopy-cube` releases before anything that
|
||||
depends on it.
|
||||
|
||||
Beyond the list: `CLAUDE.md` also needed the `typecheck` command corrected
|
||||
(`tsc --build`, not `--noEmit` — see Phase 4), a note on the vitest source alias
|
||||
and the coverage exclusion, and three entries under *Known drift*. `README.PUBLISH.md`
|
||||
absorbed the whole publish-lane rework, not just the tag prefixes: `pnpm publish`
|
||||
over `npm publish` and why, the two-pass version stamping, `verify-pack.mjs`,
|
||||
`publish-order.mjs`, `linked-deps.mjs`, and a local rehearsal recipe that uses
|
||||
**npm** to install the tarballs precisely because npm is the one that rejects a
|
||||
leaked `workspace:` range.
|
||||
|
||||
One workflow change came out of writing this up: `ci.yml` now runs
|
||||
`verify-pack.mjs` too. It was only in the two publish workflows, which means a
|
||||
leaked range would have failed the release rather than the pull request that
|
||||
introduced it — the wrong end of the process for a mistake that is free to catch
|
||||
early.
|
||||
|
||||
## Testing
|
||||
|
||||
The coverage gate (85 % branches/functions, 80 % lines/statements, per package)
|
||||
is not a CI flag — new modules without tests fail the gate locally and on the
|
||||
runner alike.
|
||||
|
||||
`tests/cubes.packages.test.ts` (new) — build a fake `node_modules` tree under
|
||||
`os.tmpdir()` and `chdir` into it, as the existing loader/config tests do:
|
||||
|
||||
- resolves a scoped and an unscoped package
|
||||
- resolves through a symlinked package directory (mimicking pnpm)
|
||||
- resolves from the declaring config's directory, not `cwd`
|
||||
- missing package → error naming the spec
|
||||
- package without `nopy.cubes` → error
|
||||
- `nopy.cubes` entry that does not exist, and one that escapes the root → errors
|
||||
- last-wins dedupe when parent and child config both name a package
|
||||
|
||||
`tests/cubes.loader.test.ts` — package-sourced cubes load; `source` attribution
|
||||
is correct for all three root types.
|
||||
|
||||
`tests/cubes.loader.edge.test.ts` — duplicate across a dir and a package errors
|
||||
and names both; cubes nested below a duplicate still get scanned (Phase 0.1);
|
||||
the error is identical regardless of scan order (Phase 0.2).
|
||||
|
||||
`tests/config.test.ts` — `cubePackages` merge, `override` resolution strategy,
|
||||
`CubePackageRef` provenance.
|
||||
|
||||
`tests/prompts.test.ts` — `coerceValue` against schemas built by a *different*
|
||||
zod instance, so Phase 0.4 cannot silently regress to `instanceof`.
|
||||
|
||||
`packages/nopy-cube/` — its own `vitest.config.ts` at the same thresholds. The
|
||||
`Manifest()` / `Manifest.create()` / `Cube.getDefaults()` cases move over from
|
||||
`tests/cubes.factories.test.ts`; what stays behind is whatever tests the
|
||||
re-export surface.
|
||||
|
||||
Resolve hook — `module.register()` is process-global, so this cannot be unit
|
||||
tested in-process. Add an integration test that spawns the CLI as a child process
|
||||
against a fixture tree, under the existing `test:integration` script.
|
||||
|
||||
## Risks
|
||||
|
||||
1. **`module.register()` is irreversible and process-wide.** It affects
|
||||
everything loaded afterwards, including the CLI's own lazy imports. Guarded
|
||||
single install, `next()`-first ordering.
|
||||
2. **Hard-error duplicates have no escape hatch.** Two bundles claiming one id
|
||||
cannot be used together, full stop. If that bites in practice the follow-up is
|
||||
a `cubeAliases` map or a per-package id prefix — explicitly out of scope here.
|
||||
3. **Store corruption.** A bundled cube writing to its own directory damages the
|
||||
pnpm global store for every project on the machine. Documented in Phase 1;
|
||||
a runtime warning is a possible follow-up.
|
||||
4. **No version compatibility check.** A bundle authored against a future `nopy`
|
||||
loaded by an older one fails at manifest-import time with a confusing error.
|
||||
A `nopy.engines` field checked at resolution time would fix it. Deferred.
|
||||
5. **Bundles vendoring cubes in their own `node_modules`** will not be found, by
|
||||
design.
|
||||
6. **`workspace:*` escaping into a published manifest.** The failure is silent at
|
||||
publish time and only shows up when someone installs the package. Phase 4
|
||||
fixes the lane; a `postpack` assertion that no dependency range starts with
|
||||
`workspace:` would make it impossible to regress.
|
||||
7. **Three packages, three version lines.** `nopy-cube` is the contract, so a
|
||||
breaking change there ripples to every published bundle in the wild — which is
|
||||
the point of versioning it separately, but it means the compatibility question
|
||||
from risk 4 gets more pressing, not less.
|
||||
@@ -51,19 +51,15 @@ Hooks can be synchronous or asynchronous (returning a `Promise`).
|
||||
|
||||
## Mechanics
|
||||
|
||||
### Sequential Execution
|
||||
### Execution Order
|
||||
|
||||
In sequential execution mode (the default), cubes added via hooks will follow the order in which they were pushed to the deployment plan:
|
||||
Cubes are always deployed sequentially, in the order they were pushed to the deployment plan. For a cube with hooks that means:
|
||||
|
||||
1. Cubes from `before` hooks.
|
||||
2. The current cube itself.
|
||||
3. Cubes from `after` hooks.
|
||||
|
||||
### Parallel Execution
|
||||
|
||||
In parallel execution mode, cubes added via hooks **do not automatically inherit dependencies**.
|
||||
|
||||
If a `before` hook calls `exec('setup-cube')`, it ensures that `setup-cube` is placed earlier in the deployment plan, but for parallel execution, you should still ensure that dependencies are correctly specified if one cube relies on another's completion.
|
||||
Because a `before` hook only places its cube *earlier in the plan*, it guarantees ordering but not much else — if the relationship is a real dependency rather than a one-off ordering nudge, declare it in `dependencies` so it is resolved and deduplicated like any other.
|
||||
|
||||
### Variable Passing
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user