4 Commits

Author SHA1 Message Date
Benjamin Diedrichsen ac050c4459 [wip] cubes packaging and distribution via registry
Publish snapshot / snapshot (push) Successful in 1m21s
2026-07-28 09:37:42 +02:00
Benjamin Diedrichsen 30d93dddc5 implementing --use-defaults 2026-07-28 09:27:05 +02:00
Benjamin Diedrichsen 5ed68c0065 improving documentation consistency. auditing documentation drifts. planning cube packaging 2026-07-27 21:58:54 +02:00
Benjamin Diedrichsen fcc181700e test release nopy-alpha5
Release / release (push) Successful in 1m0s
2026-07-27 17:07:18 +02:00
54 changed files with 2462 additions and 262 deletions
+2 -2
View File
@@ -1,7 +1,7 @@
# Every commit that lands on `main` publishes a prerelease of both packages to
# the Gitea npm registry under the `main` dist-tag:
#
# pnpm add @bitstack/nopy@main
# pnpm add @bitsquare/nopy@main
#
# The verification gate runs here rather than in ci.yml so a snapshot can never
# be published from a red `main`. Versions are derived, never committed —
@@ -87,7 +87,7 @@ jobs:
fi
install -m 600 /dev/null "$NPMRC"
{
printf '@bitstack:registry=%s\n' "$REGISTRY"
printf '@bitsquare:registry=%s\n' "$REGISTRY"
printf '//%s:_authToken=%s\n' "${REGISTRY#*://}" "$REGISTRY_TOKEN"
} >> "$NPMRC"
+3 -3
View File
@@ -9,7 +9,7 @@
# of `latest`.
#
# Required secrets:
# NPM_TOKEN npmjs granular token, read-and-write on @bitstack/*, 2FA
# NPM_TOKEN npmjs granular token, read-and-write on @bitsquare/*, 2FA
# not required. Expires after 90 days — rotate it.
# MYGITEA_NPM_TOKEN Gitea PAT with write:package. The automatic GITEA_TOKEN is
# a repo-scoped task token and the package registry rejects it.
@@ -131,7 +131,7 @@ jobs:
set -euo pipefail
install -m 600 /dev/null "$NPMRC"
{
printf '@bitstack:registry=%s\n' "$GITEA_REGISTRY"
printf '@bitsquare:registry=%s\n' "$GITEA_REGISTRY"
printf '//%s:_authToken=%s\n' "${GITEA_REGISTRY#*://}" "$GITEA_REGISTRY_TOKEN"
} >> "$NPMRC"
export npm_config_userconfig="$NPMRC"
@@ -152,7 +152,7 @@ jobs:
set -euo pipefail
install -m 600 /dev/null "$NPMRC"
{
printf '@bitstack:registry=%s\n' "$NPMJS_REGISTRY"
printf '@bitsquare:registry=%s\n' "$NPMJS_REGISTRY"
printf '//%s:_authToken=%s\n' "${NPMJS_REGISTRY#*://}" "$NPMJS_TOKEN"
} >> "$NPMRC"
export npm_config_userconfig="$NPMRC"
+152
View File
@@ -0,0 +1,152 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this repo is
A pnpm workspace holding two independently published CLIs plus the pyinfra
deployment units one of them runs:
| Path | Package | Binary | Role |
| ----------------- | ------------------- | -------- | -------------------------------------------------------- |
| `packages/nopy` | `@bitsquare/nopy` | `nopy` | interactive pyinfra script management and execution |
| `packages/keyman` | `@bitsquare/keyman` | `keyman` | SSH key management, shelling out to `age` / `ssh-keygen` |
| `cubes/` | — | — | the deployment units `nopy` runs (not published) |
The root package is private; only `packages/*` ship. The two packages do not
depend on each other.
## Commands
```sh
pnpm install # also installs the git hooks via simple-git-hooks
pnpm run build # tsc --build across both packages (project references)
pnpm run typecheck # tsc --build --noEmit
pnpm run lint # biome check . (lint:fix / lint:ci variants)
pnpm test # vitest run, both packages
pnpm run test:coverage # vitest with the coverage gate
pnpm run coverage:summary # renders the last coverage run as a Markdown table
```
Single package / single test:
```sh
pnpm --filter @bitsquare/nopy run test tests/config.test.ts # one file
pnpm --filter @bitsquare/nopy run test -t "merges configs" # by test name
pnpm --filter @bitsquare/nopy run test:watch
pnpm --filter @bitsquare/nopy run nopy # run the CLI from source via tsx
pnpm --filter @bitsquare/keyman run keyman
```
`typescript` is the 7.x native compiler, so `tsc` *is* the fast one — there is no
separate `tsgo` binary.
## Verification gate
`lint:ci``typecheck``test:coverage` is one gate, run in three places: the
`pre-push` hook, `ci.yml` (non-`main`), and `publish-snapshot.yml` (`main`).
`pre-commit` runs Biome with fixes on staged files only. Bypass with
`SKIP_SIMPLE_GIT_HOOKS=1`; re-install after editing the hook config with
`pnpm exec simple-git-hooks`.
Coverage thresholds live in each package's `vitest.config.ts` (85 % branches and
functions, 80 % lines and statements), not in a CI flag — they fail identically
locally and on the runner. Barrel files (`src/index.ts`, `src/cubes/index.ts`,
`src/nopy.cubes.ts`) and the Commander argv wiring (`src/*.cli.ts`) are excluded;
adding logic to those files means moving it somewhere covered.
Both packages set `pool: 'forks'` because tests use `process.chdir()` — most
loader/config tests build a throwaway tree under `os.tmpdir()` and chdir into it,
since discovery is driven entirely by the working directory.
## nopy architecture
One pass per invocation, `nopy.main.ts` orchestrating:
1. **`nopy.config.ts`** — `loadConfig()` walks up from `process.cwd()` collecting
every `.nopyrc.json` plus `~/.nopyrc.json`, then merges them root-first.
Per-property strategy comes from the child's `resolution` block (`merge` is
the default: arrays concatenate and dedupe, objects deep-merge; `override`
replaces). Only properties listed in `PATH_PROPERTIES` (`cubeDirs`) get
relative paths resolved against their own config file's directory. **Throws**
if no config file exists anywhere — which is why `nopy.cli.ts` calls it lazily
inside the action, so `--help`/`--version` work outside a project.
2. **`cubes/loader.ts`** — `findCubeDirectories()` unions `config.cubeDirs` with
every ancestor directory holding a `.npcubes` marker, then scans each
recursively (skipping dotted dirs and `node_modules`). A directory is a cube
when it holds both a manifest (`manifest.mjs` or `*.manifest.mjs`) and a
deploy script (`deploy.py` or `*.deploy.py`); manifests are loaded by dynamic
`import()`. Cube id = `manifest.id` → a `[id]` prefix in `manifest.name`
the directory basename. Ids are flat and need not mirror the path
(`cubes/network/tailscale` declares `net:tailscale`). Duplicate ids and bad
manifests become entries in `errors`, which aborts the run.
3. **`nopy.workflow.ts`** — picks interactive, file-replay, or history-replay and
normalises all three into a `WorkflowResult`. Replays never re-prompt except
for passwords (never persisted) and a missing host.
4. **`cubes/dependencies.ts``BuildContext.resolveCube()`** — the core.
Recursive, per (cube, host): assign params and schema defaults → collect
variables (prompt, or read them back from the session on replay) → run
`before` hooks → resolve `manifest.dependencies(vars)` (dynamic: it receives
the *collected* variables) → emit the deploy call → run `after` hooks. There
is no separate topological sort; ordering falls out of the recursion, and a
`${cubeId}:${host}` set makes emission idempotent. Hooks get a `HookContext`
whose `exec(id, vars)` re-enters `resolveCube`, so a hook can pull in a cube
that is not a declared dependency.
5. **`nopy.executor.ts`** — runs the built `pyinfra <host> -y --data K=V ... --chdir <cubeDir> <script>`
commands through execa with inherited stdio, sequentially, stopping at the
first failure unless `continueOnError`.
### Variables
`Variables` (`nopy.common.ts`) keeps three per-cube scopes plus one global bag.
`get(id)` merges them lowest-to-highest: `global` (from config `env`) →
`defaults` (Zod `.default()`, and recorded session values on replay) → `prompts`
(what the user typed) → `params` (values passed by a dependency spec or hook).
Every schema key is guaranteed present on the pyinfra side; pyinfra parses
`--data` values itself, so `"true"` arrives as a bool and numeric strings as ints.
### Cube contract
A cube directory holds `manifest.mjs` + `deploy.py`; anything else in it is
ignored by the loader but reachable from the script, which runs with the cube
directory as its cwd. Manifests are ESM, import `cubes.Manifest` from
`@bitsquare/nopy`, and declare `id`, `name`, a Zod `schema` (each field
`.describe()`d — the description is the prompt label — and `.default()`ed), plus
optional `dependencies`/`before`/`after`.
**Gotcha:** those manifests resolve `@bitsquare/nopy` through ordinary Node
resolution from the manifest's own directory. Nothing in this repo links the
package into `cubes/` or `packages/nopy/cubes/`, so loading them fails with
`ERR_MODULE_NOT_FOUND` until you link it (`pnpm --filter @bitsquare/nopy run
link:local`, then `npm link @bitsquare/nopy` where you run from).
## keyman architecture
Much smaller: `keyman.cli.ts` (argv, plus a `--print-config` escape hatch) →
`keyman.main.ts`, an inquirer menu loop dispatching to one module per operation
(`list`/`copy`/`generate`/`encrypt`/`decrypt`). `keyman.config.ts` mirrors nopy's
upward-traversal + `resolution` merge for `.keymanrc.json`, but validates the
result with Zod and falls back to defaults instead of throwing. `VAULT_ROOT` in
the environment beats the config file. Encryption shells out to `age` /
`age-keygen` / `ssh-keygen`, which must be on `PATH`.
## Releasing
Tag-driven, one package at a time; see `README.PUBLISH.md`.
- Push to `main``publish-snapshot.yml` publishes both packages to the Gitea
registry as `<version>-main.<run>.g<sha>` under the `main` dist-tag. The
version is set on the runner with `npm pkg set` and never committed.
- `git tag <dir>-v<version>` (e.g. `nopy-v1.2.0` — the directory under
`packages/`, not the npm name) → `release.yml` publishes to Gitea *and* npmjs.
The tag chooses the package, `package.json` supplies the version, and the run
fails if they disagree. A prerelease version goes out as `next`, otherwise
`latest`.
So: bump `packages/<pkg>/package.json`, land it on `main`, then tag that commit.
## Known drift
`logConfigToFlags()` is exported and tested but nothing feeds its output into the
built pyinfra command, so `log.verbosity` / `log.debug` in `.nopyrc.json`
currently have no effect. Treat `docs/REFACTORING.md` as a plan, not a record.
+732
View File
@@ -0,0 +1,732 @@
# Documentation audit
Every claim in the repository's Markdown was checked against the source it
describes. Findings are grouped by *kind of divergence*, because the fix differs:
a phantom feature needs a decision (build it or delete the docs), a wrong claim
needs an edit, a gap needs prose.
Severity is about what it costs a reader:
- **🔴 broken** — following the documentation produces a wrong result or a crash.
- **🟠 misleading** — the documentation states something the code does not do.
- **🟡 gap** — the code does something real that no document mentions.
Verified against the working tree at commit `fcc1817`. Line numbers are from that
state.
Findings closed since are marked **✅ … fixed** and keep their original text as
the record of what was wrong. So far: §1.1 (`--use-defaults`), §2.2
(`getDefaults()`), half of §2.1 (precedence), and one bullet of §6.4.
---
## Contents
- [1. Documented features that do not exist](#1-documented-features-that-do-not-exist)
- [2. Documented behaviour that differs from the code](#2-documented-behaviour-that-differs-from-the-code)
- [3. `docs/API.md` — systematic drift](#3-docsapimd--systematic-drift)
- [4. Undocumented behaviour](#4-undocumented-behaviour)
- [5. Cube documentation](#5-cube-documentation)
- [6. Defects found while verifying](#6-defects-found-while-verifying)
- [7. Checked and accurate](#7-checked-and-accurate)
- [Suggested order of attack](#suggested-order-of-attack)
---
## 1. Documented features that do not exist
These are the same class of problem as the `--parallel` flag that was removed
earlier: documented in detail, absent from the source.
### 1.1 ✅ `-D, --use-defaults` does nothing — **fixed**
> **Resolved.** The flag is now implemented; see `docs/REFACTORING.md` item 5.
> `BuildContext.resolveCube` skips the prompts, `env` in `.nopyrc.json` outranks
> the schema default so a non-interactive run can be configured, and a cube with
> a variable nothing can fill aborts the run by name instead of deploying it
> blank. The finding below is kept as the record of what was wrong.
| | |
|---|---|
| **Docs say** | `README.md:291` "Install with defaults (no prompts for customization)"; `docs/API.md:39` "Skip variable prompts, use defaults"; `nopy.cli.ts:54` "Run cubes with default values without prompts" |
| **Code does** | Nothing. |
The option is threaded through four layers and then dropped. `nopy.cli.ts:95`
`nopy.main.ts:125``nopy.main.ts:174``BuildContext.options.useDefaults`
(`cubes/dependencies.ts:35`), where it is **never read**. The only branch that
skips prompting is `isSessionReplay` (`cubes/dependencies.ts:62`).
`runInteractiveWorkflow` destructures only `useAuthKey` and ignores its
`useDefaults` too (`nopy.workflow.ts:50`).
Every documented `-D` invocation — including
`nopy install -D --save-session automated-deployment.nopysession.json`
(`README.md:226`), which is presented as the way to do an unattended run —
prompts for every variable of every cube.
```
$ grep -rn "useDefaults" packages/nopy/src/
nopy.workflow.ts:19 useDefaults?: boolean; # declared
nopy.main.ts:94 useDefaults?: boolean; # declared
nopy.main.ts:125 useDefaults = false, # defaulted
nopy.main.ts:158 { useDefaults, useAuthKey },# passed
nopy.main.ts:174 useDefaults, # passed
nopy.cli.ts:95 useDefaults: options... # passed
cubes/dependencies.ts:35 useDefaults?: boolean; # declared — and that is all
```
### 1.2 🔴 `-j, --json` produces no output on success
| | |
|---|---|
| **Docs say** | `README.md:18` "**JSON output** for CI/CD integration"; `README.md:360-367` "Machine-readable JSON output for scripting and CI/CD integration"; `docs/API.md:573` |
| **Code does** | Emits JSON **only** on failure. |
`jsonOutput` reaches three places in `nopy.main.ts` (140, 150, 219) and none of
them prints a result. It suppresses the config banner, prints
`{success: false, errors}` when cube *loading* fails, and suppresses progress
lines. The success path at `nopy.main.ts:226-236` returns the `NopyResult` object
to the caller, and `nopy.cli.ts:107-110` inspects `result.success` without
printing it.
A CI job running `nopy install --json` gets pyinfra's inherited stdio and nothing
machine-readable. The exit code is the only usable signal.
Related: `--dry-run --json` prints the **text** plan, not JSON.
`executeDeployCalls` calls `outputExecutionPlan(calls)` without the `asJson`
argument (`nopy.executor.ts:172`), even though the function supports it
(`nopy.executor.ts:110`).
### 1.3 🟠 `log.verbosity` and `log.debug` have no effect
Pre-existing known drift, recorded in `CLAUDE.md`, but the README still presents
it as a working feature — two tables, a recommendation paragraph, and a slot in
the config example (`README.md:127-130`, `143-163`).
`logConfigToFlags()` (`nopy.config.ts:352`) is exported and has 8 unit tests, but
nothing calls it. `buildDeployCall` (`cubes/dependencies.ts:105-124`) constructs
the pyinfra argv without consulting `config.log` at all.
This is live in the repo's own config: `packages/nopy/.nopyrc.json` sets
`"verbosity": "trace", "debug": true` and gets neither.
### 1.4 🟠 Manifest `env` property
`README.md:14` lists "**Default values** with optional customization via manifest
`env`". `env` was removed from `Manifest` — see `docs/REFACTORING.md` item 3, and
the current interface at `cubes/types.ts:43-56`, which has `id`, `name`,
`schema`, `dependencies`, `before`, `after` and nothing else.
### 1.5 🟠 Topological sorting
`README.md:11` ("Dependency resolution with **topological sorting**"),
`README.md:25` ("Topologically sorts cubes based on dependencies") and
`README.md:381` ("because cubes are topologically sorted") describe an algorithm
that does not exist.
There is no sort. `BuildContext.resolveCube` recurses depth-first and pushes each
cube after its dependencies, with a `${cubeId}:${host}` set for idempotence
(`cubes/dependencies.ts:43-100`, `105-143`). The ordering is a side effect of the
recursion order.
This matters beyond vocabulary: a topological sort detects cycles, and this does
not. Two cubes that depend on each other recurse until the stack overflows —
`resolvedCubes` is only consulted in `buildDeployCall`, which runs *after* the
recursive call. `docs/API.md:160` still promises `Error` "if ... circular
dependency detected".
---
## 2. Documented behaviour that differs from the code
### 2.1 🟡 Variable precedence is wrong in both directions — **partly fixed**
> **Half resolved.** `env` now outranks the Zod defaults, as documented — this
> was a prerequisite for `--use-defaults` being configurable at all. The second
> pair was deliberately left: dependency/hook params still outrank prompts. In
> practice they do not compete, because `VariableAssignment` leaves out any key
> a dependency already supplied, so the operator is never asked about it. The
> README now describes that rather than the old claim.
`README.md:107-113` stated:
> **Priority order (lowest to highest):**
> 1. Zod schema `.default()` values
> 2. Global `env` from `.nopyrc.json`
> 3. Accumulated variables from dependencies
> 4. User prompts / session replay
`Variables.get()` (`nopy.common.ts:33-39`) does:
```typescript
return {
...this.global, // 1. config env (lowest)
...this.defaults[id], // 2. Zod defaults
...this.prompts[id], // 3. what the user typed
...this.params[id], // 4. dependency / hook (highest)
};
```
Two pairs are inverted, and both have consequences:
- **Zod defaults beat `env`, not the other way round.** So `README.md:114`
"allowing users to override them globally via `.nopyrc.json`" — is backwards.
Setting `"env": {"UPDATE": false}` cannot override a cube declaring
`.default(true)`; the `env` value is only visible for keys the schema does not
define. This is why `KEY_DIR` works in `packages/nopy/.nopyrc.json` (no cube
declares it) and why anything else would not.
- **Dependency/hook parameters beat user prompts.** A value passed as
`[['user:add', {USER: 'deploy'}]]` silently overrides what the operator just
typed at the form. The docs promise the opposite.
### 2.2 ✅ "Every key ... is guaranteed to be present on `host.data`" — not when a field lacks `.default()` — **fixed**
> **Resolved.** `getDefaults()` falls back to a per-field read, so one required
> field no longer wipes out the rest; `VariableAssignment` prompts for every
> schema key rather than only the defaulted ones; and `--use-defaults` refuses
> to deploy a cube whose required key nothing supplied. Verified against all 22
> cubes in `cubes/`: 19 build a complete `-D` run, the 3 below abort by name.
`README.md:99` states it outright; `README.md:105` claims defaults ensure "every
cube has a predictable starting state".
`Cube.getDefaults()` (`cubes/types.ts:106-112`) is:
```typescript
try {
return this.manifest.schema.parse({});
} catch {
return {} as z.infer<Schema>;
}
```
One field without a `.default()` makes `parse({})` throw, and the `catch`
discards the defaults of **every other field in the cube**. `VariableAssignment`
then iterates over that empty object and returns before prompting
(`nopy.prompts.ts:175-181`), so the user is never asked. The cube deploys with no
`--data` flags at all and every `host.data.X` is `None`.
Three cubes in this repo are in that state today:
| Cube | Field(s) without `.default()` | Result |
|---|---|---|
| `net:wifi:connection` | `SSID`, `PASSWORD` | no prompt, no `--data`, all 4 vars lost |
| `service:autostart` | `APP` | no prompt, no `--data`, all 3 vars lost |
| `user:edit` | `USER` | no prompt, no `--data`, all 4 vars lost |
The failure is silent — no error, no warning, just a pyinfra run with an empty
data set.
### 2.3 🔴 `.describe()` before `.default()` loses the prompt label
`CLAUDE.md` and the cube contract state that each schema field is `.describe()`d
and "the description is the prompt label". `nopy.prompts.ts:184-185` reads it as:
```typescript
const zodType = schema[key];
const description = zodType?.description || key;
```
In zod 4, `.default()` returns a `ZodDefault` **wrapper** that does not inherit
`.description` from the type it wraps. Ordering therefore decides whether the
label survives:
```
z.boolean().describe('Update package cache').default(false) → description undefined
z.boolean().default(false).describe('Update package cache') → description preserved
```
The README's own manifest example (`README.md:67-68`) uses the losing order, so
anyone copying it gets bare `UPDATE` / `PACKAGES` keys as prompt labels instead
of the sentences they wrote. `docs/API.md:610` happens to use the working order —
the two documents disagree, and neither mentions that it matters.
15 of the 22 cubes in `cubes/` are affected; among them
`net:tailscale` (all 4 fields), `runtime:nodevm` (all 4), `user:add` (all 4),
`ssh:keygen` (all 4) and `admin:locale` (all 4).
### 2.4 🟠 Session files claim `version` and `timestamp` fields
`README.md:179-181` shows a session with `"version": "1.0.0"` and
`"timestamp": "2025-10-13T10:30:00Z"`, and `docs/SESSION_FORMAT.md:305-306`
declares both **required** in the `NopySession` interface. Every MJS example in
that file sets them.
`NopySession` (`nopy.session.ts:44-55`) has neither. `createSession`
(`nopy.session.ts:183-197`) does not add them, `saveSession` writes the object
verbatim (`nopy.session.ts:68-79`), and `loadSession`'s validation
(`nopy.session.ts:146-155`) checks only `cubes`, `hosts` and `auth`. The
repository's own `packages/nopy/example.nopysession.json` omits both — it does
not match the format its own documentation prescribes.
Consequence: a `version` field implies a compatibility check that does not exist.
Nothing reads it, so an incompatible old session fails later and more obscurely
than a version check would.
### 2.5 🟠 Session filename convention does not match `listSessions()`
The READMEs consistently use `*.nopysession.json` (`README.md:223`, `330`, `338`;
`docs/DOCKER.md:54`; the shipped `example.nopysession.json`).
`docs/SESSION_FORMAT.md` consistently uses `*.session.json` / `*.session.mjs`.
`listSessions()` (`nopy.session.ts:173`) matches only the second form:
```typescript
.filter((file) => file.endsWith('.session.json') || file.endsWith('.session.mjs'))
```
`"my-deployment.nopysession.json"` does not end in `".session.json"`, so the
file naming the README recommends is invisible to the function documented at
`docs/API.md:430`. `loadSession` is unaffected (it switches on `.json`/`.mjs`),
so this only bites the listing API.
### 2.6 🟠 `docs/DOCKER.md` container name contradicts the file it points at
`docs/DOCKER.md:35` and `:45`:
> We explicitly name it `nopy-test-container` because the
> `example.nopysession.json` is configured to target this specific container name.
`packages/nopy/example.nopysession.json` targets `@docker/nopy-test-ubuntu`
the *image* tag from the build step, not the container name. Following the guide
exactly produces a pyinfra run against a container that does not exist.
(`packages/nopy/.nopyrc.json` does list `@docker/nopy-test-container` in `hosts`,
so the guide was probably written against the config rather than the session.)
### 2.7 🟠 `docs/HOOKS.md` — hook parameters are not validated
`docs/HOOKS.md:48` describes the hook's second argument as "The final,
**validated** variables for the current cube".
`cubes/dependencies.ts:71` passes `this.variables.get(cubeId)` — a plain merge of
the four scopes. `schema.parse()` is called in exactly one place,
`Cube.getDefaults()` on an empty object, to extract defaults. Values from
prompts, `env`, dependencies and hooks are never validated against the schema at
any point in the pipeline. `coerceValue` (`nopy.prompts.ts:145-159`) type-coerces
prompt input, which is not the same as validation and does not apply to the other
three scopes.
### 2.8 🟠 `docs/HOOKS.md` — dependencies can pass variables too
The comparison table at `docs/HOOKS.md:83` says variable passing is
"Inherited from env" for dependencies versus "Explicitly passed via `exec()`" for
hooks, presenting explicit parameters as a hook-only capability.
`DependencySpec` is `string | [id, variables?]` (`cubes/types.ts:23`) and
`cubes/dependencies.ts:86-88` unpacks the tuple and forwards it into the same
`params` scope that `exec()` writes to. The two mechanisms are identical in this
respect; per §2.1 both outrank user prompts.
### 2.9 🟠 nopy README installation section describes the wrong package manager
`README.md:248-279` says "This package is part of a **yarn** workspace monorepo",
then gives `yarn install`, `yarn workspace @bitsquare/nopy build`,
`yarn workspace @bitsquare/nopy nopy`, and `yarn nopy`.
The repo is a **pnpm** workspace: `packageManager: "pnpm@11.17.0"` in the root
manifest, `pnpm-workspace.yaml`, a `pnpm-lock.yaml`, and every other document
(root `README.md`, `README.PUBLISH.md`, `CLAUDE.md`) uses pnpm. There is no
`yarn.lock`.
The section is also the wrong content for the file. `README.md` is one of three
files shipped in the npm tarball (`files: ["dist", "README.md", "LICENSE"]`), so
this is what a reader sees on npmjs.com — build-from-monorepo instructions
instead of `npm install -g @bitsquare/nopy`, which is what the root README and
`README.PUBLISH.md:314` correctly tell people to run.
### 2.10 🟠 keyman README: two operations missing, one operation invented
`packages/keyman/README.md:90-96` lists four menu entries: List, Encrypt,
Decrypt, Quit. The menu (`keyman.main.ts:54-61`) has six:
```
📋 List keys 📝 Copy public key 🆕 Generate key
🔒 Encrypt keys 🔓 Decrypt keys ❌ Quit
```
`Copy public key` and `Generate key` are undocumented — the latter being the only
way to create a key inside the tool, which is why the Quick Start
(`packages/keyman/README.md:33`) tells the user to shell out to `ssh-keygen`
manually.
Conversely `packages/keyman/README.md:11` advertises "🔄 Support for key
rotation". There is no rotation anywhere: `grep -rn "rotat" packages/keyman/src/`
returns nothing.
`packages/keyman/README.md:93` also says encrypt takes keys "from `vault/tmp/`".
`encryptKeys` (`keyman.encrypt.ts:12-22`) unions `~/.ssh` and `vault/tmp`, and
offers both in the checkbox.
### 2.11 🟡 Root README understates the coverage gate
Root `README.md:57-58` describes "a hard **85 % branch** floor". Both
`vitest.config.ts` files set four thresholds: branches 85, functions 85, lines
80, statements 80. `README.PUBLISH.md:135` and `CLAUDE.md` both state all four —
the root README is the odd one out, and it is the file a new contributor reads
first.
### 2.12 🟡 `docs/DOCKER.md` relative link is broken
`docs/DOCKER.md:8` links `[README.md](./README.md)`, which resolves to
`packages/nopy/docs/README.md` — nonexistent. It should be `../README.md`.
---
## 3. `docs/API.md` — systematic drift
`docs/API.md` documents an earlier architecture. It is not a matter of
individual stale lines: the two central type definitions, one whole module, and
two of the documented functions describe code that no longer exists. Anyone
building against this file writes code that will not compile.
Recommendation: regenerate rather than patch.
### 3.1 🔴 Functions that do not exist
| Documented | Reality |
|---|---|
| `resolveDependencies(cubes, selectedCubeNames)` (`API.md:142-160`) | No such export. Resolution is `BuildContext.resolveCube` and returns nothing — it accumulates into `deployCalls`. |
| `buildDeployCalls(cubeNames, hosts, context)` (`API.md:286-313`) | No such export. The entire "Builder Module" section, and its `BuildResult` interface, describes code replaced by `BuildContext` (`docs/REFACTORING.md` item 2). Still listed in the table of contents at `API.md:12`. |
### 3.2 🔴 `Cube<Schema>` — wrong shape entirely
`API.md:75-84` documents an interface with `key`, `dependencies: string[]`,
`schema`, `defaults()`, `before`, `after`.
`Cube` (`cubes/types.ts:88-113`) is a **class**: constructor `(manifest, dir,
deployScript)`, getters `id` and `name`, method `getDefaults()`. Everything else
lives behind `.manifest`. Not one documented member name is correct — `key` is
`id`, `defaults()` is `getDefaults()`, and `dependencies`/`schema`/`before`/
`after` are on `cube.manifest`, not on `cube`.
### 3.3 🔴 `Manifest<Schema>` — wrong shape
`API.md:92-100` documents `key`, `dependencies: string[]`, `defaults: () => ...`.
`Manifest` (`cubes/types.ts:43-56`) has `id` (not `key`), no `defaults` member at
all, and `dependencies` is a **function of the collected variables**:
```typescript
dependencies?: (variables: z.infer<Schema>) => DependencySpec[];
```
That signature change is the headline of `docs/REFACTORING.md` item 2. The
`API.md` example at `:171` does use the function form, so the file contradicts
itself two paragraphs apart. Both `Cube` and `Manifest` are additionally shown
as generic over `z.AnyZodObject`, which zod 4 removed; the codebase defines
`AnyObjectSchema` for exactly this reason (`cubes/types.ts:13`).
### 3.4 🟠 Incorrect signatures and examples
| Location | Documented | Actual |
|---|---|---|
| `API.md:486-492` | `saveConfig(data, local?)`, example passes `false` | `saveConfig(data, configPath?: string)` (`nopy.config.ts:323`). Passing `false` writes nothing useful. |
| `API.md:481-484` | Search order `./nopyrc.json` then `~/.nopyrc.json` | Filename is `.nopyrc.json` (leading dot). Home is applied **first** (lowest priority, `nopy.config.ts:117-120`), all ancestors are collected and merged root-first, and the function **throws** when none is found (`nopy.config.ts:285-289`). The `resolution` merge strategy is not mentioned. |
| `API.md:534-540` | `VariableAssignment(cube, env)` returning vars | `VariableAssignment(cube, variables: Variables)` returns `Promise<void>` and mutates the `Variables` instance (`nopy.prompts.ts:167`). The example's return value is always `undefined`. |
| `API.md:321-331` | `runWorkflow(sessionPath, cubes, config, options?)` | Takes a fifth parameter `replaySession?: NopySession` (`nopy.workflow.ts:206-212`) — the entire history-replay path. |
| `API.md:337-344` | `WorkflowResult.cubesWithDependencies` | Field is `selectedCubes` (`nopy.workflow.ts:31`). |
| `API.md:202-209` | `DeployCall.dependencies: string[]` | `DependencySpec[]` (`nopy.executor.ts:27`) — and always `[]` in practice (`cubes/dependencies.ts:132`). |
| `API.md:452-457` | `NopyConfig` with 4 fields | Missing `history` and `execution` (`nopy.config.ts:68-81`). |
| `API.md:37-45` | 7 `NopyOptions` parameters | Missing `printOnly`, `replaySession`, `saveToHistory` (`nopy.main.ts:93-104`). |
| `API.md:169-175` | `cubes.Manifest` example | Omits `id`, the field that determines the cube's identity. |
### 3.5 🟡 Exported and undocumented
Public API in `src/index.ts` with no `API.md` entry: the entire history module
(`addToHistory`, `listHistory`, `getLastSession`, `getSessionById`,
`clearHistory`, `removeFromHistory`, `loadHistory`, `saveHistory`,
`getHistoryPath`, `formatHistoryList`, `HISTORY_FILE`, `DEFAULT_HISTORY_SIZE`,
plus `HistoryEntry` / `SessionHistory`), `BuildContext`,
`runSessionReplayWorkflow`, `getConfigPaths`, `findCubeDirectories`, `getCube`,
`filterInternalVariables`, `separateEnvAndCubeVariables`.
The CLI cheat-sheet (`API.md:554-578`) omits `-R`, `-H`, `-P`, `--no-history`,
and the `history` / `clear-history` commands.
`ManifestFactory` (`cubes/factories.ts:28`) is marked `@deprecated` but is not
re-exported from `cubes/index.ts`, so it is unreachable dead code.
---
## 4. Undocumented behaviour
### 4.1 🔴 Cubes in this repo cannot be loaded
`CLAUDE.md` records the `@bitsquare/nopy` linking gotcha; the package README does
not mention it at all, and the gotcha is incomplete.
Cube manifests are loaded by dynamic `import()` from their own directory
(`cubes/loader.ts:75`), so they resolve their imports through ordinary Node
resolution from `cubes/…`. Nothing in the tree provides either dependency:
```
node_modules/@bitsquare/ → absent
packages/nopy/node_modules/@bitsquare/ → absent
cubes/node_modules/ → absent
```
`@bitsquare/nopy` is the documented half. **`zod` is the other half** — 20 of 22
manifests `import { z } from 'zod'`, and that fails independently of the nopy
link. Verified by loading every manifest with a resolver hook: with only
`@bitsquare/nopy` mapped, 20 of 22 fail `ERR_MODULE_NOT_FOUND: zod`.
Because `loadCubes` turns each failure into an `errors` entry and `nopy.main.ts:147-152`
aborts when `errors.length > 0`, a fresh clone cannot run a single cube. Neither
README mentions a setup step.
### 4.2 🟠 The SSH password is printed in plaintext
Not stated in any document, and it sits directly against the security notes at
`README.md:217` and `:325` (which are narrowly about *storage*, and are correct
as far as they go).
`buildDeployCall` embeds the password in the command string
(`cubes/dependencies.ts:112-113`):
```typescript
parts.push(`--user ${this.auth.username} --password ${this.auth.password}`);
```
That string is then:
1. logged at debug level — `log.debug(\`Command: ${commandStr}\`)`
(`nopy.executor.ts:76`) — and the `nopy` logger is configured with
`lowestLevel: 'debug'` (`nopy.main.ts:41-44`), so it **prints to the console
on every deployment**;
2. printed unmasked by `--dry-run``outputExecutionPlan` masks values whose key
contains "password" in the `--data` section (`nopy.executor.ts:134`) but prints
`call.command.join(' ')` verbatim one line earlier (`nopy.executor.ts:127`);
3. passed through `execa({shell: true})` (`nopy.executor.ts:79`), making it
visible in the process list and, without quoting, vulnerable to shell
metacharacters in the password.
### 4.3 🟠 History and session files record only prompted values
`README.md:414` says an entry records "the variable values that were answered at
the prompts" — accurate, but the consequence is not drawn out.
`buildDeployCall` records `this.variables.get(cubeId, 'prompts')`
(`cubes/dependencies.ts:138`) — the prompts scope alone. Values that came from a
dependency spec, a hook's `exec()`, `env`, or a schema default are **not** in the
entry.
Combined with §2.1 (dependency params outrank prompts) this means a replay can
legitimately produce a different command than the run it replays, if the
dependency graph resolved differently.
### 4.4 🟡 `-P, --print-only` is undocumented
Implemented (`nopy.cli.ts:61`, `nopy.main.ts:202-213`), listed in the CLI's own
help examples (`nopy.cli.ts:38`), and absent from `README.md` and `docs/API.md`.
It prints the built pyinfra commands grouped by cube and exits.
Worth documenting alongside `--dry-run`, since the difference is not obvious:
`--print-only` returns a `NopyResult` with `successful: 0` and skips execution
entirely, while `--dry-run` goes through the executor.
### 4.5 🟡 `--save-session` is ignored during a replay
`nopy.main.ts:191` guards with `saveSessionPath && !workflow.isReplay`, so
`nopy install -R -s out.json` writes nothing and says nothing. The
"Recording a Session" section (`README.md:219-227`) does not mention it.
### 4.6 🟡 `.npcubes` is documented but unused in this repo
`README.md:43` shows a `cubes/.npcubes` marker in the layout diagram and
`README.md:244` documents the mechanism. `find . -name .npcubes` returns nothing
— discovery here runs entirely off `cubeDirs` in `.nopyrc.json`. The feature
exists in `findCubeDirectories` (`cubes/loader.ts:26-30`); the diagram just shows
a file that no reader will find if they go looking.
### 4.7 🟡 `ssh:keyman` depends on a global `env` value
`cubes/ssh/keyman/deploy.py` reads `host.data.get('KEY_DIR')`, which no manifest
declares — it comes from `env` in `.nopyrc.json`. This works (§2.1: `env` is
visible for keys the schema does not define) and it is the only cube relying on
the mechanism, but nothing documents the coupling. Anyone running that cube from
a project without `KEY_DIR` in their config gets `None`.
---
## 5. Cube documentation
Two cubes have **no README at all**: `cubes/admin/hostname` and `cubes/git/clone`
(20 of 22 have one).
### 5.1 🔴 `cubes/service/autostart/README.md` documents a different cube
The file is titled **"TypeStack Install Cube"** and describes cloning a git
repository, `yarn install`, `yarn build`, `docker compose up -d`, and PM2 process
management. The manifest (`service:autostart`, "Manage systemd service
autostart") does none of that — it has three fields and calls `systemd.service`.
| README documents | In the schema? |
|---|---|
| `USER` | ❌ |
| `REPO` | ❌ |
| `ENV` | ❌ |
| `NODE_PATH` | ❌ |
| `APP` | ✅ |
| `AUTOSTART` | ✅ |
| — | `SERVICE_NAME` (undocumented) |
Every "Requirements" entry (Git, Yarn, Docker, PM2, NVM, SSH keys) is inapplicable.
It reads as a leftover from a cube that was split or renamed.
### 5.2 🟠 `cubes/network/wifi/access-point/README.md` — four wrong parameters
| README | Manifest |
|---|---|
| `NETWORK_DEVICE` (default `wlan0`) | does not exist |
| `CHANNEL` | does not exist — **but `deploy.py` reads it** (see §6.3) |
| `IP_ADDRESS` (default `192.168.50.1`) | field is `AP_IP`, default `192.168.4.1` |
| `CONNECTION_NAME` default `net:wifi:ap` | default is `pi-point` |
| SSID / PASSWORD listed as "Required" | both have defaults (`PiPoint` / `1223334444`) |
Both worked examples set keys that will be ignored.
### 5.3 🟠 Two cubes claim to have no parameters
| Cube | README says | Schema has |
|---|---|---|
| `cubes/runtime/docker` | "This cube currently has no configurable parameters." | `DISTRO` (`ubuntu` \| `debian`) |
| `cubes/runtime/nodevm` | "This cube currently has no configurable parameters." | `VERSION`, `USER`, `ALIAS`, `GLOBAL_PACKAGES` |
`nodevm`'s README also describes installing "the latest LTS via the official
NodeSource setup script", while the manifest is named "Install nvm and nodejs
with global packages" and takes an explicit `VERSION`.
(`admin/cockpit` and `armor/fail2ban` make the same claim and are correct — both
have empty schemas.)
### 5.4 🟡 Cube id conventions are inconsistent
- `cubes/caddy/base` declares `id: 'caddy'` while its sibling declares
`caddy:spa`. Every other nested cube uses the `group:name` form.
- `net:wifi:connection` sets `name: 'network:wifi:connection - Connect to a WiFi
network'` and `user:edit` sets `name: 'user:edit - Modify an existing user
account'` — the id is baked into the display name. Since the picker renders
`${cube.id} - ${cube.name}` (`nopy.prompts.ts:43`), these show as
`net:wifi:connection - network:wifi:connection - Connect to a WiFi network`.
`README.md:54` documents `[id]`-in-name as a *fallback* for a missing `id`
field, not as a prefix to carry alongside one.
---
## 6. Defects found while verifying
Not documentation issues, but found while checking the docs and worth recording.
### 6.1 🔴 `cubes/service/autostart/deploy.py` cannot run
```python
from pyinfra.operations import systemd # `server` is never imported
from pyinfra import host
APP = host.data.APP # AUTOSTART and SERVICE_NAME never read
if AUTOSTART: # NameError
...
server.shell(...) # NameError, in the else branch
```
`AUTOSTART` and `SERVICE_NAME` are declared in the manifest and never pulled off
`host.data`; `server` is used but not imported. The script raises `NameError` on
the `if`. Per §2.2 this cube also gets no `--data` at all, so it fails twice over.
### 6.2 🔴 `-H <id>` and `--no-history` share one destination
Both options write to `options.history` (`nopy.cli.ts:57` and `:64`). Verified
with Commander:
```
argv=[] -> {} # undefined → saves
argv=["--no-history"] -> {"history":false} # correct
argv=["-H","abc123"] -> {"history":"abc123"} # correct
argv=["-H","abc123","--no-history"] -> {"history":false} # id destroyed
```
In the last case the `-H` argument is silently discarded and nopy falls through
to a full interactive run instead of replaying. `saveToHistory: options.history
!== false` (`nopy.cli.ts:104`) works only because the two meanings happen not to
collide in the common cases.
### 6.3 🟠 `access-point/deploy.py` reads an undeclared variable
`host.data.get('CHANNEL')` — no manifest declares `CHANNEL`, so it is always
`None`. The README documents it as a supported optional parameter (§5.2). One of
the three has to give.
### 6.4 🟡 Debug output left in the shipped code
- ~~`nopy.common.ts:22` — `console.log('Assigning', artefactId, scope, values)`
fires on every variable assignment, printing values to the console. Combined
with §4.2 this is a second path by which secrets reach stdout.~~ **Removed**
alongside the `--use-defaults` work; it would have made an unattended run
unreadable. The two other paths in §4.2 are untouched.
- `keyman.encrypt.ts:19-20` — `console.log(tmpKeys); console.log(sshKeys);`
before the prompt.
### 6.5 🟡 No cycle detection
Covered under §1.5. `docs/API.md:160` documents the error; there is no code that
raises it. Mutually dependent cubes recurse until the stack overflows.
---
## 7. Checked and accurate
Recording what was verified and found correct, so a future pass need not redo it.
- **`README.PUBLISH.md`** — checked against `.gitea/workflows/*.yml` and both
manifests. Workflow triggers, the `files` array, dist-tag rules, the snapshot
version format, `upload-artifact@v3`, `cache@v4`, the `npm pack --dry-run`
step, `retention-days: 7`, and all four coverage thresholds are right. The
only file that states the coverage gate completely.
- **Root `README.md`** — pnpm/corepack, Node ≥ 22 with `.nvmrc` pinning 24, the
script table, and both git hooks match the root `package.json`. Only the
coverage line is incomplete (§2.11).
- **`CLAUDE.md`** — accurate throughout, including the `logConfigToFlags` drift
note and the resolution/merge description. Two additions worth making: `zod` is
missing from the cube-linking gotcha (§4.1), and `-D` being a no-op (§1.1)
belongs under "Known drift".
- **`README.md` history section** (`:392-429`) — the recording rules, the
fail-then-`-R` flow, replay-does-not-re-record, the four suppression cases, the
`defaults`-layer replay semantics, and the `Cube not found` failure mode all
check out against `nopy.history.ts`, `nopy.main.ts:195-200` and
`cubes/dependencies.ts:62-69`.
- **`README.md` continue-on-error section** (`:369-390`) — fail-fast, no
rollback, skipped-cubes-absent-from-results, exit code 1, and CLI-over-config
precedence match `nopy.executor.ts:180-193` and `nopy.cli.ts:67-68`.
- **`README.md` cube layout and discovery** — the directory-pair rule, recursive
scan, dotted/`node_modules` skipping, the prefixed `*.manifest.mjs` fallback,
and the three-step id resolution match `cubes/loader.ts` exactly.
- **pyinfra `--data` type coercion** (`README.md:101`) — correct.
- **keyman config** — priority (`VAULT_ROOT` > file > defaults), the four default
values, and the vault layout match `keyman.config.ts` and `keyman.encrypt.ts`.
---
## Suggested order of attack
**1 — ~~Decide on the three phantom features.~~ Two left.** §1.1 (`-D`) is
**done** — implemented, tested, and verified against every cube in `cubes/`.
That closed §2.2 and half of §2.1 with it, since neither could be left standing
under a run that never prompts. §1.2 (`--json`) and §1.3 (`log.*`) are still
"documented, wired up, never read": each is a small implementation or a small
deletion, but neither can stay documented as working.
**4 — Decide the `.describe()`/`.default()` ordering (§2.3).** Either read
through the `ZodDefault` wrapper in `nopy.prompts.ts`, or fix the ordering in all
14 manifests and the README example. The first is one line and cannot regress.
**5 — Regenerate `docs/API.md` (§3).** Too far gone to patch: two core types,
one whole module, and two functions describe code that no longer exists.
**6 — Cube docs (§5) and the two missing READMEs.** `service/autostart` is the
worst — its README belongs to a different cube, and its `deploy.py` does not run
at all (§6.1).
**7 — Secrets on stdout (§4.2, §6.4).** The `console.log` in `Variables.assign`
is gone. Still open: mask the password in the executor's debug line and in the
dry-run plan, and pass `--user`/`--password` as argv rather than interpolating
into a shell string.
+19 -19
View File
@@ -23,8 +23,8 @@ shipped. If you only want to cut a release, jump to
| Directory | Package | Binary |
| ----------------- | ------------------ | -------- |
| `packages/nopy` | `@bitstack/nopy` | `nopy` |
| `packages/keyman` | `@bitstack/keyman` | `keyman` |
| `packages/nopy` | `@bitsquare/nopy` | `nopy` |
| `packages/keyman` | `@bitsquare/keyman` | `keyman` |
Both are ESM, both declare `engines.node >= 22`, and both expose a single
executable through `bin`, so `npm install -g` puts `nopy` / `keyman` on the
@@ -155,7 +155,7 @@ semver even when the abbreviated sha happens to be all digits. The run number is
monotonic, so every push produces a version that has never existed before.
```sh
pnpm add @bitstack/nopy@main
pnpm add @bitsquare/nopy@main
```
Snapshots never reach npmjs and never move `latest`. The version is written into
@@ -175,7 +175,7 @@ edit is discarded with the workspace and is never committed.
```
The tag name is `<directory>-v<version>` — the directory under `packages/`, not
the npm name. `nopy-v1.2.0`, not `@bitstack/nopy-v1.2.0`.
the npm name. `nopy-v1.2.0`, not `@bitsquare/nopy-v1.2.0`.
The tag decides **which** package ships; `package.json` decides the **version**.
The workflow re-reads the manifest and refuses to continue if the two disagree:
@@ -193,7 +193,7 @@ a coincidence, not a requirement.
What a successful run leaves behind:
- `@bitstack/<pkg>@<version>` on the Gitea registry
- `@bitsquare/<pkg>@<version>` on the Gitea registry
- the same tarball on npmjs, public, under `latest` or `next`
- a Gitea release on the tag, with notes and an install snippet
- a step summary with both install commands
@@ -228,7 +228,7 @@ organisation to share across repos.
| Secret | Required | Purpose |
| ----------------- | -------- | --------------------------------------------------------- |
| `NPM_TOKEN` | yes | npmjs granular token, read-and-write on `@bitstack/*` |
| `NPM_TOKEN` | yes | npmjs granular token, read-and-write on `@bitsquare/*` |
| `MYGITEA_NPM_TOKEN` | yes | Gitea PAT with `write:package` |
`GITEA_TOKEN` is injected into every run by Gitea itself, and the workflows fall
@@ -241,12 +241,12 @@ practice. Create it under **Settings → Applications → Access Tokens** with t
`package` scope set to read-and-write; its owner needs package-write on the
`BitSquare` organisation, since the registry path is org-owned.
For npmjs, create a **granular access token** scoped to `@bitstack/*` with
For npmjs, create a **granular access token** scoped to `@bitsquare/*` with
read-and-write permission, and set 2FA to not-required so it works
unattended. npm warns against that combination and points at Trusted Publishing
instead — but Trusted Publishing federates only GitHub Actions and GitLab CI/CD
over OIDC, and Gitea is not a provider it accepts. A token is the only route
from this runner. Scoping the token to `@bitstack/*` is what keeps the exposure
from this runner. Scoping the token to `@bitsquare/*` is what keeps the exposure
small: a leak lets someone publish to that scope, not touch the account.
> npm caps granular token lifetime at 90 days, so `NPM_TOKEN` needs rotating
@@ -264,7 +264,7 @@ small: a leak lets someone publish to that scope, not touch the account.
## Registry authentication in the workflows
`release.yml` has to talk to two different registries about the same `@bitstack`
`release.yml` has to talk to two different registries about the same `@bitsquare`
scope inside one job. It does that without ever mutating `~/.npmrc`:
- each publish step writes its own credentials file, created with
@@ -284,20 +284,20 @@ file written into the workspace can never be committed by accident.
From npmjs — public, no configuration:
```sh
npm install -g @bitstack/nopy @bitstack/keyman
npm install -g @bitsquare/nopy @bitsquare/keyman
```
From the Gitea registry, which holds every snapshot plus a mirror of every
release. Per-project, in the repo's `.npmrc`:
```ini
@bitstack:registry=https://gitea.bitsquare.dev/api/packages/BitSquare/npm/
@bitsquare:registry=https://gitea.bitsquare.dev/api/packages/BitSquare/npm/
```
Globally with credentials, in `~/.npmrc`:
```ini
@bitstack:registry=https://gitea.bitsquare.dev/api/packages/BitSquare/npm/
@bitsquare:registry=https://gitea.bitsquare.dev/api/packages/BitSquare/npm/
//gitea.bitsquare.dev/api/packages/BitSquare/npm/:_authToken=<your gitea token>
```
@@ -308,7 +308,7 @@ instance and organisation automatically.
To track snapshots in another project:
```sh
pnpm add @bitstack/nopy@main
pnpm add @bitsquare/nopy@main
```
## Design decisions
@@ -360,14 +360,14 @@ Try the binary as an end user would get it, without publishing:
```sh
cd packages/nopy && pnpm run link:local # build + npm link
nopy --help
npm unlink -g @bitstack/nopy
npm unlink -g @bitsquare/nopy
```
Check that a version is not already taken before you tag:
```sh
npm view @bitstack/nopy@1.2.0 version # npmjs
npm view @bitstack/nopy@1.2.0 version \
npm view @bitsquare/nopy@1.2.0 version # npmjs
npm view @bitsquare/nopy@1.2.0 version \
--registry https://gitea.bitsquare.dev/api/packages/BitSquare/npm/
```
@@ -391,14 +391,14 @@ npm view @bitstack/nopy@1.2.0 version \
Meanwhile:
```sh
npm dist-tag add @bitstack/nopy@1.1.9 latest # point users back
npm deprecate @bitstack/nopy@1.2.0 "Broken build, use 1.2.1"
npm dist-tag add @bitsquare/nopy@1.1.9 latest # point users back
npm deprecate @bitsquare/nopy@1.2.0 "Broken build, use 1.2.1"
```
`npm unpublish` is only possible within 72 hours and burns the version number
forever; a deprecation with a working `latest` is almost always the better move.
**On Gitea**, delete the version under **Packages → @bitstack/… → Settings**
**On Gitea**, delete the version under **Packages → @bitsquare/… → Settings**
before that exact version can be published again.
**A bad tag** can be moved, but only before the release workflow has published
+4 -4
View File
@@ -5,12 +5,12 @@ they deploy.
| Path | Package | Binary | What it is |
| ----------------- | ------------------ | -------- | --------------------------------------------------- |
| `packages/nopy` | `@bitstack/nopy` | `nopy` | interactive pyinfra script management and execution |
| `packages/keyman` | `@bitstack/keyman` | `keyman` | SSH key management with `age` encryption |
| `packages/nopy` | `@bitsquare/nopy` | `nopy` | interactive pyinfra script management and execution |
| `packages/keyman` | `@bitsquare/keyman` | `keyman` | SSH key management with `age` encryption |
| `cubes/` | — | — | the deployment units `nopy` runs |
```sh
npm install -g @bitstack/nopy @bitstack/keyman
npm install -g @bitsquare/nopy @bitsquare/keyman
```
See each package's README for usage, and
@@ -37,7 +37,7 @@ pnpm install
`typescript` is on the 7.x native compiler, so `tsc` *is* the fast one — there is
no separate `tsgo` binary to keep in sync. Each package also has a dev-run script
(`pnpm --filter @bitstack/nopy run nopy`) that executes the TypeScript sources
(`pnpm --filter @bitsquare/nopy run nopy`) that executes the TypeScript sources
directly through `tsx`.
## Git hooks
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
export default cubes.Manifest({
id: 'admin:cockpit',
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes, uniqid } from '@bitstack/nopy';
import { cubes, uniqid } from '@bitsquare/nopy';
import { z } from 'zod';
/**
+1 -1
View File
@@ -12,7 +12,7 @@ Configures system keyboard layout permanently by updating `/etc/default/keyboard
## Usage
```javascript
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
export default cubes.Manifest({
name: 'My Host Setup',
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
export default cubes.Manifest({
id: 'armor:fail2ban',
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
// [agnt://cogen/cogen/network-wifi-connection-1]{cartridge: "ansiblings/cubes", action: "generated", status: "generated"}
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
+1 -1
View File
@@ -1,4 +1,4 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
// [agnt://cogen/cogen/user-edit-1]{cartridge: "ansiblings/cubes", action: "generated", status: "generated"}
+1 -1
View File
@@ -1,5 +1,5 @@
{
"name": "@bitstack/keyman",
"name": "@bitsquare/keyman",
"version": "1.0.0",
"description": "A system to simplify ssh key management",
"keywords": [
+170 -75
View File
@@ -6,21 +6,58 @@ A CLI tool that simplifies **pyinfra** script management and execution, providin
Nopy wraps pyinfra with structure, validation, and an interactive experience for managing complex infrastructure deployments. It organizes deployments into self-contained "cubes" with dependency management, schema validation, and lifecycle hooks.
## Features
- **Dependency resolution** with topological sorting
- **Before/after hooks** for multi-cube orchestration
- **SSH key or password authentication**
- **Default values** with optional customization via manifest `env`
- **Schema validation** using Zod
- **Recursive cube directory discovery**
- **Dry-run mode** for previewing deployments
- **JSON output** for CI/CD integration
- **Session history** with replay capability
## Workflow
1. **Load cubes** - Discovers and validates cubes from configured directories
2. **Interactive prompts** - Select cubes, target host, and authentication method
3. **Dependency resolution** - Topologically sorts cubes based on dependencies
4. **Variable assignment** - Validates and collects configuration with schema validation
5. **Execute hooks** - Runs before/after hooks for orchestration
6. **Deploy** - Sequentially executes pyinfra commands
## Core Concepts
### Cubes
Self-contained deployment units consisting of:
A cube is a **directory** containing two files:
- **Python deployment script**: `<cube-name>.deploy.py`
- **JavaScript manifest**: `<cube-name>.manifest.mjs` defining schema, dependencies, defaults, and hooks
- **Configuration variables**: Validated with Zod schemas
- **JavaScript manifest**: `manifest.mjs` defining schema, dependencies, defaults, and hooks
- **Python deployment script**: `deploy.py`, a plain pyinfra script
Configuration variables are declared in the manifest and validated with Zod schemas before the deployment script runs.
```
cubes/
├── .npcubes
└── apt/
└── install/
├── manifest.mjs
└── deploy.py
```
Any directory holding both files is treated as a cube, so cubes can be nested as deeply as you like to group them by topic. Discovery is recursive; directories starting with `.` and `node_modules` are skipped. Additional files in the cube directory (a `README.md`, config templates, and so on) are ignored by the loader and can be referenced from the deploy script — the script runs with its cube directory as the working directory.
The prefixed forms `<cube-name>.manifest.mjs` and `<cube-name>.deploy.py` are also still recognized, but plain `manifest.mjs` / `deploy.py` is the current convention.
A cube's identity comes from the manifest's `id` field (see below). If `id` is omitted, nopy falls back to an `[id]` prefix in the manifest `name`, and finally to the directory's own name. Note that the id does not have to mirror the folder path — `cubes/network/tailscale` declares `id: 'net:tailscale'`.
#### Cube Manifest
```javascript
import { z } from 'zod'
import { cubes } from '@bitstack/nopy'
import { cubes } from '@bitsquare/nopy'
export default cubes.Manifest({
id: 'apt:install',
@@ -33,6 +70,36 @@ export default cubes.Manifest({
})
```
#### Deployment Script
The matching `deploy.py` is a plain pyinfra script. Nopy passes each schema variable to pyinfra as `--data KEY=value`, so they are available on `host.data`:
```python
from pyinfra import host
from pyinfra.operations import apt
UPDATE = host.data.UPDATE
PACKAGES = str(host.data.PACKAGES).split(' ')
apt.packages(
name='Install essential packages',
packages=['ca-certificates', 'gnupg', 'lsb-release'],
update=UPDATE,
_sudo=True,
)
apt.packages(
name='Install custom packages',
packages=[p.strip() for p in PACKAGES if p],
update=UPDATE,
_sudo=True,
)
```
Every key defined in the manifest `schema` is guaranteed to be present on `host.data` — either from the Zod `.default()`, from `.nopyrc.json`, from a dependency, or from a user prompt.
**Value types**: pyinfra parses `--data` values before your script sees them. `"true"` / `"false"` become booleans, numeric strings become `int`, valid JSON becomes the parsed structure, and everything else stays a string. This is why `UPDATE` can be handed straight to pyinfra's `update=` argument, while `PACKAGES` is wrapped in `str(...)` before splitting.
#### Variable Defaults
Variable defaults are defined directly in the Zod schema using `.default()`. This ensures that every cube has a predictable starting state and provides type-safe default values.
@@ -41,10 +108,14 @@ Variable defaults are defined directly in the Zod schema using `.default()`. Thi
1. Zod schema `.default()` values
2. Global `env` from `.nopyrc.json`
3. Accumulated variables from dependencies
4. User prompts / session replay
3. User prompts, or the recorded answers on session replay
4. Variables passed in by a dependency or a hook
This allows cubes to ship with reasonable defaults while still allowing users to override them globally via `.nopyrc.json` or interactively during deployment.
This allows cubes to ship with reasonable defaults while still allowing users to override them globally via `.nopyrc.json` or interactively during deployment. Because `env` outranks the schema, `.nopyrc.json` is also what steers a run started with `--use-defaults`, which never prompts.
3 and 4 rarely compete: a key a dependency supplies is left out of the prompt entirely, so the user is only ever asked about the keys nothing else has set.
A field declared without `.default()` has none of sources 1 and 2 to fall back on. It is prompted for like any other, with an empty initial value — but a run that cannot prompt (`--use-defaults`) fails on it unless `env` or a dependency provides it.
### Configuration
@@ -60,10 +131,19 @@ Uses `.nopyrc.json` files (project-level or home directory) containing:
"log": {
"verbosity": "info",
"debug": false
},
"history": {
"maxSessions": 10,
"autoSave": true
},
"execution": {
"continueOnError": false
}
}
```
`history` controls automatic session recording (see [Deployment History](#deployment-history)), and `execution.continueOnError` sets the default for `--continue-on-error`.
#### Logging Configuration
Control pyinfra output verbosity and debug information using the `log` configuration object:
@@ -84,39 +164,6 @@ Control pyinfra output verbosity and debug information using the `log` configura
| `false` | (none) | No debug logs (default) | Normal operation |
| `true` | `--debug` | Enable pyinfra debug logs | Deep debugging of pyinfra internals |
**Examples:**
Basic troubleshooting:
```json
{
"log": {
"verbosity": "info"
}
}
```
Debug command failures:
```json
{
"log": {
"verbosity": "trace"
}
}
```
Deep debugging with pyinfra internals:
```json
{
"log": {
"verbosity": "trace",
"debug": true
}
}
```
**Recommendation:** Start with `"info"` for typical troubleshooting, use `"trace"` when investigating command failures, and enable `debug: true` only when debugging pyinfra itself.
### Session Recording and Replay
@@ -209,7 +256,7 @@ This package is part of a yarn workspace monorepo. Install from the repository r
```bash
# From repository root (/ansiblings)
yarn install
yarn workspace @bitstack/nopy build
yarn workspace @bitsquare/nopy build
```
To use the `nopy` command globally, you can:
@@ -217,7 +264,7 @@ To use the `nopy` command globally, you can:
1. **Use yarn workspace command**:
```bash
yarn workspace @bitstack/nopy nopy
yarn workspace @bitsquare/nopy nopy
```
2. **Link the package globally**:
@@ -253,6 +300,25 @@ nopy install --use-defaults
nopy install -D
```
Skips the per-cube variable form. Every variable is taken from the sources that
need no interaction — the Zod `.default()`, `env` in `.nopyrc.json`, and values
handed over by a dependency or a hook — which is what makes `.nopyrc.json` the
place to configure an unattended run.
Cube selection, host and authentication are still asked for; there is nowhere
else for them to come from. Pair `-D` with `-K` to skip the auth question too,
or with `-R` / `-H` / `-l`, which supply all three from the recorded session.
A cube whose schema declares a field with **no** `.default()` cannot be filled in
this way, so the run stops before anything is deployed rather than passing the
variable as empty:
```
Error: Cube "net:wifi:connection" cannot run with --use-defaults: SSID, PASSWORD
have no default values. Set them under "env" in .nopyrc.json, pass them from a
dependency, or drop --use-defaults to be prompted.
```
**Use SSH key authentication**:
```bash
@@ -264,11 +330,23 @@ nopy install -K
**Repeat last run**:
```bash
nopy install --repeat-last-run
nopy install --repeat-last
# or
nopy install -R
```
Every deployment is automatically recorded to a `.nopy.history.json` file in the current working directory, so the last run is always available to `-R` without having to pass `--save-session` first. The default retention is the 10 most recent sessions (configurable via `history.maxSessions`); use `nopy history` to list them and `nopy install -H <id>` to replay any one of them — see [Deployment History](#deployment-history).
The recording happens before the deploy commands run, so a **failed** deployment is recorded too — `-R` is the quick way to retry one after fixing the cause. Replaying a session with `-R` or `-H` does not itself create a new entry, so repeating never pushes the original run out of the list.
A run is *not* recorded when:
- `--dry-run` or `--no-history` is passed
- No cubes were selected, so there was nothing to deploy
- `history.autoSave` is set to `false` in `.nopyrc.json`
Because the history file is resolved against the current working directory, each project keeps its own history — running nopy from a different directory will not find the previous run. As with session files, passwords are never stored and are re-prompted on replay.
**Save session for replay**:
```bash
@@ -302,14 +380,6 @@ nopy install --dry-run
Shows the execution plan including commands, environment variables, and targets without running anything. Sensitive data is masked in output.
**Parallel execution**:
```bash
nopy install --parallel
```
Executes independent cubes in parallel using a dependency graph. Cubes are grouped into execution stages, with a default concurrency limit of 4.
**JSON output (for CI/CD)**:
```bash
@@ -323,17 +393,64 @@ Machine-readable JSON output for scripting and CI/CD integration.
```bash
nopy install --continue-on-error
# or
nopy install -c
```
Continue deploying remaining cubes even if one fails.
**View deployment history**:
**Default behaviour (fail-fast)**: without this flag, nopy stops at the first cube that fails. Cubes are deployed sequentially in dependency order, so the failing cube's output is the last thing you see — every cube still queued behind it is skipped entirely and is never attempted.
This is deliberate: because cubes are topologically sorted, a cube that fails is often a dependency of the ones after it, and continuing would deploy them onto a half-configured host.
Two consequences worth knowing:
- **Cubes that already succeeded are not rolled back.** The host is left in a partial state — the cubes before the failure are applied, the rest are not. Fix the cause and re-run; well-written cubes are idempotent, so re-applying the earlier ones is normally harmless.
- **Skipped cubes are not reported as failed.** They are simply absent from the results, so a summary of "3 successful, 1 failed" out of 6 cubes means the remaining 2 were never run.
Either way, the command exits with code `1` if any cube failed, which is what CI picks up. Use `--continue-on-error` when your cubes are genuinely independent and you would rather collect every failure in one run than stop at the first.
The default can be flipped for a project by setting `execution.continueOnError` in `.nopyrc.json`; the CLI flag takes precedence over it.
#### Deployment History
```bash
nopy history # List recent deployments
nopy history --json # Same list as JSON, including each recorded session
nopy install -H <id> # Replay a specific deployment by ID
nopy clear-history # Delete all recorded sessions
```
History is what makes [Repeat last run](#basic-commands) work, but it holds more than just the last deployment: every recorded run stays replayable until newer runs push it out. `nopy history` (alias `nopy h`) lists them newest first, with `` marking the entry that `-R` would replay:
```
Session History:
→ [1] 07/26/2026, 14:32 - apt:install, net:tailscale → root@web-01
ID: mdk3n1qx4a2fh
[2] 07/26/2026, 09:05 - apt:install → root@web-01
ID: mdk0zzp8b71cq
Total: 2 session(s)
```
Each entry records the selected cubes together with the variable values that were answered at the prompts, the target hosts, the authentication method, and the username — never the password. Pass an ID to `-H` to run that exact combination again:
```bash
nopy install -H mdk0zzp8b71cq
```
A replay is non-interactive: cube selection, host, and variable values all come from the entry, so nopy runs straight through without asking anything. The two exceptions are password authentication, which always re-prompts, and an entry with no recorded host, which falls back to the host picker.
Two things are worth knowing before relying on an older entry:
- **Recorded values are applied as defaults, not as a frozen snapshot.** If a cube's schema has gained a variable since the run was recorded, the replay neither prompts for it nor fails — the new variable quietly takes its Zod `.default()`. Global `env` values are likewise read from the *current* `.nopyrc.json` rather than from the entry.
- **A replay fails if a cube no longer exists.** Renaming or deleting a cube id makes every history entry that referenced it unreplayable: nopy logs `Cube from session not found` and then aborts with `Cube not found: <id>`.
The history lives in `.nopy.history.json` in the working directory and uses the same structure as a session file, so trimming the array by hand is a perfectly good way to prune it. It does contain the variable values that were entered, which is why it is listed in this repository's `.gitignore` — treat it like any other file holding deployment configuration. A corrupt or unreadable history file is treated as empty rather than raising an error, which looks exactly like a project that has never been deployed from.
For a run you want to keep indefinitely, don't rely on history — it rotates. Use `--save-session` to write it to a file you control (see [Session Recording and Replay](#session-recording-and-replay)).
### Development
**Run without building**:
@@ -348,28 +465,6 @@ npm run nopy
npm run debug
```
## Workflow
1. **Load cubes** - Discovers and validates cubes from configured directories
2. **Interactive prompts** - Select cubes, target host, and authentication method
3. **Dependency resolution** - Topologically sorts cubes based on dependencies
4. **Variable assignment** - Validates and collects configuration with schema validation
5. **Execute hooks** - Runs before/after hooks for orchestration
6. **Deploy** - Sequentially executes pyinfra commands
## Features
- **Dependency resolution** with topological sorting
- **Parallel execution** of independent cubes in stages
- **Before/after hooks** for multi-cube orchestration
- **SSH key or password authentication**
- **Default values** with optional customization via manifest `env`
- **Schema validation** using Zod
- **Recursive cube directory discovery**
- **Dry-run mode** for previewing deployments
- **JSON output** for CI/CD integration
- **Session history** with replay capability
## Documentation
- [Cube Hooks](docs/HOOKS.md) - Lifecycle hooks for dynamic orchestration
+3 -3
View File
@@ -1,6 +1,6 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
export default cubes.Manifest({
name: '[apt-all] Test dependencies',
dependencies: () => ['apt/more'],
name: '[test:apt-all] Test dependencies',
dependencies: () => ['test:apt-more'],
});
@@ -1,8 +1,8 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default cubes.Manifest({
name: '[apt:essentials] Install essential packages',
name: '[test:apt-essentials] Install essential packages',
dependencies: () => [],
schema: z.object({
UPDATE: z.boolean().default(false),
+3 -3
View File
@@ -1,6 +1,6 @@
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
export default cubes.Manifest({
name: '[apt-more] Test dependencies',
dependencies: () => [['apt/essentials']],
name: '[test:apt-more] Test dependencies',
dependencies: () => [['test:apt-essentials']],
});
+22 -30
View File
@@ -24,7 +24,7 @@ This document describes the public API for the nopy package.
Main entry point for nopy deployments.
```typescript
import { nopy } from '@bitstack/nopy';
import { nopy } from '@bitsquare/nopy';
const result = await nopy({
useDefaults: false,
@@ -41,7 +41,6 @@ const result = await nopy({
| `saveSession` | `string` | - | Path to save session file |
| `loadSession` | `string` | - | Path to load session for replay |
| `dryRun` | `boolean` | `false` | Show execution plan without running |
| `parallel` | `boolean` | `false` | Execute independent cubes in parallel |
| `continueOnError` | `boolean` | `false` | Continue after failures |
| `jsonOutput` | `boolean` | `false` | Output results as JSON |
@@ -87,7 +86,7 @@ interface Cube<Schema extends z.AnyZodObject = z.AnyZodObject> {
#### `Manifest<Schema>`
Cube manifest (used in `*.manifest.mjs` files).
Cube manifest (used in `manifest.mjs` files).
```typescript
interface Manifest<Schema extends z.AnyZodObject = z.AnyZodObject> {
@@ -160,23 +159,14 @@ const order = resolveDependencies(cubes, ['apt-all']);
**Throws:** `Error` if cube not found or circular dependency detected
#### `buildExecutionStages(cubes, selectedCubeNames)`
Groups cubes into stages for parallel execution.
```typescript
const stages = buildExecutionStages(cubes, ['apt-all', 'docker']);
// Returns: [['apt:essentials'], ['apt-more', 'docker'], ['apt-all']]
```
**Returns:** `string[][]` - Array of stages
#### `createManifest(options)`
#### `cubes.Manifest(options)`
Factory function for creating cube manifests.
```typescript
export default createManifest({
import { cubes } from '@bitsquare/nopy';
export default cubes.Manifest({
name: 'My Cube',
dependencies: () => [['apt:essentials']],
schema: z.object({
@@ -185,6 +175,8 @@ export default createManifest({
});
```
`createManifest` and `manifest` are exported as equivalent aliases; `cubes.Manifest` is the documented form.
#### `uniqid(length?)`
Generates a random alphanumeric string.
@@ -239,8 +231,6 @@ Options for deployment execution.
```typescript
interface ExecutionOptions {
parallel?: boolean;
concurrency?: number;
continueOnError?: boolean;
dryRun?: boolean;
onProgress?: (result: ExecutionResult, completed: number, total: number) => void;
@@ -252,12 +242,11 @@ interface ExecutionOptions {
#### `executeDeployCalls(calls, options?)`
Executes an array of deployment calls.
Executes an array of deployment calls sequentially, in the order they were built.
```typescript
const results = await executeDeployCalls(calls, {
parallel: true,
concurrency: 4,
continueOnError: false,
onProgress: (result, completed, total) => {
console.log(`${completed}/${total}`);
},
@@ -581,9 +570,6 @@ nopy install -l ./my-session.json
# Dry run
nopy install -n
# Parallel execution
nopy install -p
# JSON output
nopy install -j
@@ -597,21 +583,27 @@ nopy install -c
### File Structure
A cube is a directory containing both a `manifest.mjs` and a `deploy.py`:
```
cubes/
└── my-cube/
├── my-cube.manifest.mjs
└── my-cube.deploy.py
├── manifest.mjs
└── deploy.py
```
Cube directories may be nested for grouping (`cubes/apt/install/`), and any extra files alongside the pair are available to the deploy script via relative paths.
The prefixed forms `<cube-name>.manifest.mjs` and `<cube-name>.deploy.py` are still recognized for backwards compatibility.
### Manifest Example
```javascript
// my-cube.manifest.mjs
import { createManifest } from '@bitstack/nopy';
// manifest.mjs
import { cubes } from '@bitsquare/nopy';
import { z } from 'zod';
export default createManifest({
export default cubes.Manifest({
name: 'My Cube',
dependencies: () => [['apt:essentials']],
schema: z.object({
@@ -634,7 +626,7 @@ export default createManifest({
### Deploy Script Example
```python
# my-cube.deploy.py
# deploy.py
from pyinfra import host
from pyinfra.operations import apt, server
+568
View File
@@ -0,0 +1,568 @@
# Cube bundles as npm packages
Status: **Phase 0 has landed; Phases 16 are still a plan, not a record.**
Distributing cubes as npm packages so a project can `pnpm add @acme/cubes-net`
and have its cubes show up in `nopy` alongside local ones.
## Goals
- A cube bundle is an ordinary npm package, publishable to npmjs or Gitea
through the existing release lanes.
- A consuming project opts into a bundle explicitly, by name, in `.nopyrc.json`.
- Existing manifests, dependency specs (`dependencies: () => ['apt:essentials']`)
and stored session history keep working untouched.
- The in-repo `cubes/` tree becomes the first published bundle, proving the path.
## Non-goals
- Automatic discovery of bundles from the dependency tree. Cubes run privileged
deploy scripts against real hosts; a transitive dependency contributing one
silently is a supply-chain hole. Opt-in per package, always.
- Namespacing or id rewriting. Ids stay flat and global (see *Decisions*).
- Version compatibility checks between a bundle and the `nopy` running it.
Noted as a risk, deferred.
## Decisions
| Question | Decision |
| --- | --- |
| Duplicate cube ids across sources | **Hard error.** No precedence, no shadowing. Mitigation is a good error message, not a fallback. |
| Id format | Unchanged, flat. The id is the session key (`dependencies.ts:135`); changing it breaks `--repeat-last` and `--history`. |
| Discovery | Explicit `cubePackages` list in `.nopyrc.json`. |
| Migrate in-repo `cubes/` | Yes — `packages/cubes-core`, as the proof of concept. |
| Split an authoring package (`@bitsquare/nopy-cube`) | **Yes.** Bundles take a regular dependency on it; `@bitsquare/nopy` re-exports it for backwards compatibility. See *Phase 4*. |
## Current state
What already works, unchanged:
- `--chdir <cubeDir>` (`nopy.executor.ts`) means a `deploy.py` under
`node_modules` runs fine; pyinfra only needs the path.
- `scanDirectory` skips `node_modules` when *descending* (`loader.ts:101`), not
for the root it is handed. So `"cubeDirs": ["./node_modules/@acme/cubes-net/cubes"]`
works today. That is the escape hatch until this lands, and it stays working
afterwards.
What blocks a clean story:
1. **Manifest imports.** `manifest.mjs` does `import { cubes } from '@bitsquare/nopy'`,
resolved by ordinary Node resolution from the manifest's own directory. From
inside `node_modules/@acme/cubes-net/`, that resolves upward into the
consumer's `node_modules` — fine if the consumer installed `@bitsquare/nopy`,
`ERR_MODULE_NOT_FOUND` if `nopy` is only installed globally. Same gotcha
CLAUDE.md already documents for the local `cubes/` tree.
2. **No way to name a package** in config, only paths.
3. **Recursively scanning `node_modules` is not a workaround.** pnpm symlinks
direct deps, and `readdir(withFileTypes)` reports a symlink as
`isSymbolicLink()`, not `isDirectory()` — the scan would skip every package.
Package roots must be resolved explicitly.
## Phase 0 — fixes that land first — **done**
Independent of packaging, and the duplicate-id work depends on them.
**0.1 `scanDirectory` drops subtrees on duplicates.** `loader.ts:84-87` pushed
the error and `return`ed, which exited before the recursive descent at line 100.
Cubes nested below a duplicate never got scanned, so the error report was
incomplete: you fix one collision, re-run, find the next.
**0.2 Duplicate detection is order-dependent.** `loadCubes()` ran `Promise.all`
over folders into a shared `cubes` object, so which source was "first" and which
was "the duplicate" varied run to run.
Both are one restructure. Scanning and id resolution are now separate passes:
each root fills its own `ScanResult`, the lists are concatenated in root order
(`Promise.all` preserves input order regardless of completion order), and a
grouping pass builds `cubes` and the errors. `scanDirectory` no longer decides
anything about ids, so it always descends. Directory entries are sorted, and a
directory reachable from two roots is deduped by path — one cube seen twice is
not a collision, which it used to be reported as.
**0.3 `apt:essentials` is already declared twice.** `cubes/apt/essentials`
declares it via `id`; `packages/nopy/cubes/apt/essentials` declared it via an
`[apt:essentials]` prefix in `name`. `cubeDirs` merges root-first, so running
`nopy` from `packages/nopy` collected both and aborted. Confirmed against the
real trees before the rename:
```
Duplicate cube id 'apt:essentials' from 2 sources:
/…/ansiblingz/cubes/apt/essentials
/…/ansiblingz/packages/nopy/cubes/apt/essentials
Rename one of them, or remove a source from .nopyrc.json.
```
The three `packages/nopy/cubes` fixtures are now `[test:apt-essentials]`,
`[test:apt-all]` and `[test:apt-more]`; all 25 cubes load with no errors. Their
`dependencies` were stale too — they named `apt/more` and `apt/essentials`,
which are not ids anything declares — so they now point at the renamed ids.
**0.4 `coerceValue` breaks if zod is ever duplicated.** `nopy.prompts.ts:147-154`
discriminated with `instanceof z.ZodDefault`, `z.ZodBoolean`, `z.ZodNumber` and
friends — checks against the *running CLI's* zod instance. The moment a bundle
resolves its own copy of zod (entirely possible once manifests arrive from
`node_modules`; see Phase 4), every check returns false and `coerceValue` falls
through to the raw string, silently. Booleans stop being booleans.
`defaultValueOf` in `cubes/types.ts` had the same breakage, reached whenever a
schema has one field without a `.default()``getDefaults()` tries
`safeParse({})` first, which is instance-agnostic, and only then drops to the
per-field read.
Both now discriminate on `def.type`, a plain string that holds across instances,
via two exported helpers (`zodKind`, `zodInner`). Verified on the installed zod
4.4.3:
```
z.boolean().default(false).def.type → 'default'
z.boolean().default(false).def.innerType → { def: { type: 'boolean' } }
z.number().def.type → 'number'
```
`tests/helpers/foreign-zod.ts` rebuilds a schema as plain objects carrying zod's
`def` but not its prototype — structurally what a second copy of zod produces,
and `instanceof`-blind, so neither call site can regress.
Worth noting for Phase 4: zod 4 exposes `def.defaultValue` as a getter that
already invokes a lazily declared default, so the `typeof === 'function'` branch
in `defaultValueOf` is now dead. It is kept as insurance against that changing.
## Phase 1 — the bundle contract
A cube bundle is an npm package with a `nopy` field:
```json
{
"name": "@acme/cubes-net",
"version": "1.0.0",
"type": "module",
"nopy": { "cubes": ["./cubes"] },
"files": ["cubes", "README.md", "LICENSE"],
"keywords": ["nopy", "nopy-cubes", "pyinfra"],
"dependencies": {
"@bitsquare/nopy-cube": "^1.0.0",
"zod": "^4.4.3"
},
"publishConfig": { "access": "public" }
}
```
Rules:
- `nopy.cubes` — directories relative to the package root, scanned exactly like
`cubeDirs` entries. Required; a package listed in `cubePackages` without a
`nopy` field is an error, not a silent skip. Listing it means the user expects
cubes from it.
- Both dependencies are **regular dependencies, not peers**, and both are
load-bearing: a manifest imports `Manifest` from `@bitsquare/nopy-cube` and `z`
from `zod`. `@bitsquare/nopy-cube` peer-depends on zod, so the bundle's copy is
the one everybody uses — see Phase 4.
- The package needs no `exports` entry for this to work — resolution reads
`package.json` off disk (Phase 2), so the `exports` map is irrelevant.
- **A bundle's directory is read-only at runtime.** Under pnpm, `node_modules`
content is hardlinked into the global store; a cube writing next to its own
`deploy.py` corrupts that store for every project on the machine. Cubes must
write to `/tmp` or the remote host, never their own dir.
- A bundle must not ship a `.nopyrc.json`. Config discovery walks up from
`process.cwd()`, never from cube directories, so it would never be read.
## Phase 2 — resolution
### Config surface
```json
{
"cubePackages": ["@acme/cubes-net", "@acme/cubes-caddy"]
}
```
Merges through the existing `resolution` machinery for free — arrays concat and
dedupe — so a parent config supplies the org baseline and a child adds to it.
**Resolution origin.** A package must be resolved from *the directory of the
config file that declared it*, not from `process.cwd()`. Otherwise a bundle
listed in `~/.nopyrc.json` cannot resolve unless every project happens to depend
on it. This is the same problem `PATH_PROPERTIES` solves for `cubeDirs`, but the
output is a tagged reference rather than a rewritten string:
```ts
export interface CubePackageRef {
spec: string; // '@acme/cubes-net'
from: string; // dirname of the .nopyrc.json that declared it
}
```
So the file format and the loaded format diverge for this one key:
```ts
interface NopyConfigFile extends Omit<Partial<NopyConfig>, 'cubePackages'> {
cubePackages?: string[];
resolution?: ResolutionConfig;
}
interface NopyConfig {
cubePackages: CubePackageRef[];
// ...
}
```
`resolveConfigPaths()` performs the `string → CubePackageRef` conversion, next to
where it resolves `PATH_PROPERTIES`. Two consequences to handle:
- `mergeValue`'s array dedupe only fires when every element is a primitive
(`config.ts:142`), so refs fall through to plain concat. Dedupe by `spec` in
the resolver instead.
- Dedupe is **last-wins**: merge order is root-first, so the last occurrence is
the most specific config, and its `from` is the right resolution origin.
### Resolver
New file `packages/nopy/src/cubes/packages.ts`:
```ts
export interface CubePackage {
name: string;
root: string;
dirs: string[]; // absolute, from nopy.cubes
}
export function resolveCubePackages(
refs: CubePackageRef[]
): { packages: CubePackage[]; errors: string[] };
```
Locate the package root without going through `exports` and without tripping on
pnpm symlinks:
```ts
const req = createRequire(path.join(ref.from, 'noop.js'));
for (const dir of req.resolve.paths(ref.spec) ?? []) {
const manifest = path.join(dir, ref.spec, 'package.json');
if (fs.existsSync(manifest)) return path.dirname(manifest);
}
```
`resolve.paths()` walks the `node_modules` chain upward from `ref.from` plus the
global paths; `existsSync` follows symlinks, so pnpm's
`node_modules/@acme/cubes-net → ../.pnpm/…` resolves correctly.
Errors (each aborts the run, consistent with the existing `errors` contract):
- package not found on any candidate path
- `package.json` unparseable
- no `nopy.cubes`, or it is not a non-empty array of strings
- a `nopy.cubes` entry escapes the package root, or does not exist
### Wiring
`findCubeDirectories()` currently returns `string[]`. It becomes the union of
three sources, each tagged so the loader can attribute a cube to it:
```ts
export type CubeRoot =
| { type: 'dir'; dir: string } // cubeDirs, .npcubes markers
| { type: 'package'; dir: string; packageName: string }; // cubePackages
export function findCubeRoots(): { roots: CubeRoot[]; errors: string[] };
```
Keep `findCubeDirectories()` as a thin wrapper returning `roots.map(r => r.dir)`
it is exported from `src/cubes/index.ts` and covered by tests. The
`node_modules` skip inside `scanDirectory` stays and is now *correct*: a
bundle's own `node_modules` should not be scanned.
## Phase 3 — hard errors with attribution
`Cube` gains a source, as an optional fourth constructor parameter so the public
signature stays backwards compatible:
```ts
export type CubeSource =
| { type: 'dir'; dir: string }
| { type: 'package'; packageName: string; dir: string };
class Cube {
constructor(
manifest: Manifest<Schema>,
dir: string,
deployScript: string,
source: CubeSource = { type: 'dir', dir }
) {}
}
```
The duplicate error carries both sources and is order-independent (Phase 0.2):
```
Duplicate cube id 'apt:essentials' from 2 sources:
package @bitsquare/cubes-core /…/node_modules/@bitsquare/cubes-core/cubes/apt/essentials
directory /repo/packages/nopy/cubes/apt/essentials
Rename one of them, or remove a source from .nopyrc.json.
```
There is deliberately no override, alias or precedence rule. If two bundles ever
claim the same id they are mutually exclusive, and the fix is upstream.
Surface the source in the interactive picker and in `--json` output so a user can
see where a cube came from before running it.
## Phase 4 — `@bitsquare/nopy-cube`, the authoring package
The problem: a manifest does `import { cubes } from '@bitsquare/nopy'`, resolved
by ordinary Node resolution from the manifest's own directory. From inside
`node_modules/@acme/cubes-net/`, that only resolves if the consumer installed
`@bitsquare/nopy` locally — a globally-installed CLI leaves nothing to find.
The fix is to give bundles something they can depend on *normally*, so resolution
is plain, boring, spec-compliant Node with no loader tricks in the critical path.
### The package
`packages/nopy-cube` — the `Manifest` factory, the `Cube` class, and the types
from `cubes/types.ts`. No CLI, no `execa`, `inquirer`, `enquirer`, `zx`, or
`commander`. Today a cube manifest — a file that ships nothing but data — drags
the entire CLI in as a transitive dependency; this makes the authoring surface
honest about how small it is, and gives the *contract* a version number that
moves independently of the CLI's.
```json
{
"name": "@bitsquare/nopy-cube",
"version": "1.0.0-alpha0",
"type": "module",
"exports": { ".": { "types": "./dist/index.d.ts", "default": "./dist/index.js" } },
"peerDependencies": { "zod": "^4.4.3" },
"files": ["dist", "README.md", "LICENSE"]
}
```
**zod is a peer, deliberately.** Bundles declare zod as a regular dependency, so
exactly one zod instance serves the manifest, the schema it builds, and the
`Manifest` factory. Phase 0.4 removes the CLI's `instanceof` dependence on that
being the *same* copy the CLI uses, but keeping the bundle side single-instance
is still the right default.
### Moving `cubes/types.ts`
`@bitsquare/nopy` re-exports everything from `@bitsquare/nopy-cube` — through
`src/cubes/index.ts` and the `cubes` namespace in `src/nopy.cubes.ts`, both
already coverage-excluded barrels — so `import { cubes } from '@bitsquare/nopy'`
in every existing manifest keeps working unchanged. Nothing in `cubes/` has to be
touched at migration time.
Repo plumbing this requires:
- `tsconfig.base.json`: add `"@bitsquare/nopy-cube": ["./packages/nopy-cube/src"]`
to `paths`.
- Root `tsconfig.json`: add the project reference.
- `packages/nopy/tsconfig.json`: `references` is currently `[]` — add
`{ "path": "../nopy-cube" }`. This is the first reference edge in the repo, so
`tsc --build` ordering starts mattering.
- `packages/nopy/package.json`: `"@bitsquare/nopy-cube": "workspace:*"`.
- A `vitest.config.ts` for the new package with the same thresholds. `Manifest()`,
`Manifest.create()` and `Cube.getDefaults()` all carry logic, so the relevant
cases move over from `tests/cubes.factories.test.ts`.
### The release lane needs fixing first
This is the part that is easy to miss. `link-workspace-packages` is unset and
pnpm 10+ defaults it to `false`, so a plain semver range would resolve
`@bitsquare/nopy-cube` from the registry instead of linking the workspace copy —
the dependency has to use `workspace:*`.
But **both workflows publish with `npm publish`**, and npm does not understand
the `workspace:` protocol. `@bitsquare/nopy` would ship a manifest carrying
`"@bitsquare/nopy-cube": "workspace:*"`, which fails on install with
`EUNSUPPORTEDPROTOCOL`. This has never mattered because the two current packages
do not depend on each other; `nopy → nopy-cube` is the first edge, and the PoC
bundle in Phase 5 adds a second.
Pick one before publishing anything:
- **Switch to `pnpm publish --no-git-checks`**, which rewrites the protocol to a
concrete version on pack. Cleanest, but changes the publish step in both
workflows and pulls in pnpm's own lifecycle behaviour.
- **Rewrite the range with `npm pkg set` before publishing**, extending the
pattern `publish-snapshot.yml` already uses for `version`. In `release.yml` one
package ships at a time, so it pins to whatever version `packages/nopy-cube/package.json`
declares at that commit. In `publish-snapshot.yml` the loop needs to become two
passes — compute every snapshot version first, then publish — so `nopy` can pin
the exact `nopy-cube` snapshot from the same run.
Recommendation: `pnpm publish`, and verify against the Gitea registry with a
throwaway version before the first real release.
### Also: the resolve hook
Independent of the split, and worth building anyway — it retires the
`ERR_MODULE_NOT_FOUND` gotcha CLAUDE.md documents for the local `cubes/` tree,
where manifests import `@bitsquare/nopy` from a directory that has no link to it.
With the split, the hook is a convenience rather than load-bearing: bundles
resolve `@bitsquare/nopy-cube` through their own `node_modules` and never reach
it.
**The gotcha is bigger than CLAUDE.md says: it is two specifiers, not one.**
Measured by linking `@bitsquare/nopy` into the root `node_modules` and loading
the real tree — every manifest then failed on `Cannot find package 'zod'`
instead. Manifests import `z` directly to build their schema, and pnpm's
isolated layout puts zod under `packages/nopy/node_modules`, not the root. A
hook that only covers `@bitsquare/nopy` moves the error rather than fixing it,
so it has to fall back for `zod` too. With both linked, all 25 cubes load.
Falling back for `zod` hands local cubes the *CLI's* zod instance, so no
duplication arises there. Bundles are the case that duplicates it, and Phase 0.4
is what makes that safe.
New `packages/nopy/src/nopy.resolve-hook.mjs`, registered once from `loadCubes()`
before the first `import(manifestPath)`:
```ts
module.register('./nopy.resolve-hook.mjs', import.meta.url, {
data: { fallback: import.meta.resolve('./index.js') },
});
```
The hook tries `next(specifier, ctx)` **first** and only falls back to the
running CLI's own copy on failure. That ordering matters: a consumer that has its
own `@bitsquare/nopy` installed keeps using it, so the hook never silently
introduces version skew.
Constraints:
- `module.register()` is process-global and cannot be undone. Install it once,
behind a module-level guard.
- The hook file runs on a separate thread; the `data` payload must be
structured-cloneable (a string URL is).
- The `.mjs` must ship in `dist` and be listed in `files` — it already is, via
the `dist` entry.
- It resolves `@bitsquare/nopy` and `zod`, not `@bitsquare/nopy-cube`. Bundles
never depend on the hook; only the in-repo `cubes/` tree and hand-written local
cubes do.
## Phase 5 — proof of concept: `packages/cubes-core`
Depends on Phase 4 shipping first — the bundle cannot declare
`@bitsquare/nopy-cube` as a dependency until it exists, and the publish-lane fix
has to be in place before either package is published.
1. `git mv cubes packages/cubes-core/cubes` — preserves per-file history.
2. Add `packages/cubes-core/package.json` per the Phase 1 contract. Version
`1.0.0-alpha0`, tracking the current alpha train. Not private. Its
`@bitsquare/nopy-cube` dependency uses `workspace:*` in the repo, which is
exactly the case the Phase 4 publish fix has to handle.
Migrating the manifests' `import { cubes } from '@bitsquare/nopy'` to
`import { Manifest } from '@bitsquare/nopy-cube'` is optional — the re-export
keeps the old form working — but doing it here is what proves the bundle
resolves without the CLI present at all.
3. Root `.nopyrc.json`: **replace** `"cubeDirs": ["./cubes"]` with
`"cubePackages": ["@bitsquare/cubes-core"]`. Replace, not add — keeping both
means every id resolves from two sources and the hard error fires on every
run.
4. Root `package.json`: add `"@bitsquare/cubes-core": "workspace:*"` to
`devDependencies`, so pnpm symlinks it into the root `node_modules`. This is
what makes the PoC exercise the real pnpm symlink resolution path rather than
a plain directory.
5. `packages/nopy/.nopyrc.json` keeps `"cubeDirs": ["./cubes"]` for its fixtures.
Config merges root-first, so running from `packages/nopy` now pulls in
`@bitsquare/cubes-core` *and* the fixtures — which is exactly the collision
Phase 0.3 renames away.
6. Workflow changes are limited to the publish-lane fix from Phase 4.
`publish-snapshot.yml` loops `for dir in packages/*/` and picks both new
packages up automatically; `release.yml` resolves `packages/<pkg>` from the
tag, so `cubes-core-v1.0.0` and `nopy-cube-v1.0.0` work as-is. Verify on the
first snapshot run that a package with no `build` script is skipped cleanly by
`pnpm -r run build` (it is) and that publishing is happy with no lifecycle
scripts.
7. No `tsconfig` reference for `cubes-core` — the bundle has no TypeScript. (The
`nopy-cube` references from Phase 4 are separate.)
8. Biome already lints `cubes/**/*.mjs` from the root; only the path changes.
### Verifying the PoC
- **In-workspace:** `pnpm --filter @bitsquare/nopy run nopy -P` from the repo
root lists `net:tailscale`, `apt:install`, … and prints deploy commands whose
`--chdir` points into `node_modules/@bitsquare/cubes-core/cubes/…`.
- **Out-of-workspace (the real test):** `npm pack` the bundle, install the
tarball into a throwaway directory with a `.nopyrc.json` naming it, install
`nopy` *globally*, and run `nopy -P`. This is what actually exercises Phase 4 —
a manifest resolving its import from a `node_modules` tree that has no
`@bitsquare/nopy` in it. Check the installed tarball's `package.json` really
carries a concrete `@bitsquare/nopy-cube` range and not `workspace:*`.
## Phase 6 — documentation
- `CLAUDE.md`: the repo table gains two rows (`packages/nopy-cube`,
`packages/cubes-core`) and loses the `cubes/` one; "The two packages do not
depend on each other" is no longer true; the loader section in *nopy
architecture*; and the *Gotcha* paragraph, which the resolve hook retires.
- `packages/nopy/docs/CUBE-BUNDLES.md` (new): authoring guide — package shape,
read-only constraint, id collision policy, publishing.
- `packages/nopy/docs/API.md` + `README.md`: `cubePackages`.
- `README.PUBLISH.md`: `nopy-cube-v*` and `cubes-core-v*` as new tag prefixes,
plus the ordering constraint — `nopy-cube` releases before anything that
depends on it.
## Testing
The coverage gate (85 % branches/functions, 80 % lines/statements, per package)
is not a CI flag — new modules without tests fail the gate locally and on the
runner alike.
`tests/cubes.packages.test.ts` (new) — build a fake `node_modules` tree under
`os.tmpdir()` and `chdir` into it, as the existing loader/config tests do:
- resolves a scoped and an unscoped package
- resolves through a symlinked package directory (mimicking pnpm)
- resolves from the declaring config's directory, not `cwd`
- missing package → error naming the spec
- package without `nopy.cubes` → error
- `nopy.cubes` entry that does not exist, and one that escapes the root → errors
- last-wins dedupe when parent and child config both name a package
`tests/cubes.loader.test.ts` — package-sourced cubes load; `source` attribution
is correct for all three root types.
`tests/cubes.loader.edge.test.ts` — duplicate across a dir and a package errors
and names both; cubes nested below a duplicate still get scanned (Phase 0.1);
the error is identical regardless of scan order (Phase 0.2).
`tests/config.test.ts``cubePackages` merge, `override` resolution strategy,
`CubePackageRef` provenance.
`tests/prompts.test.ts``coerceValue` against schemas built by a *different*
zod instance, so Phase 0.4 cannot silently regress to `instanceof`.
`packages/nopy-cube/` — its own `vitest.config.ts` at the same thresholds. The
`Manifest()` / `Manifest.create()` / `Cube.getDefaults()` cases move over from
`tests/cubes.factories.test.ts`; what stays behind is whatever tests the
re-export surface.
Resolve hook — `module.register()` is process-global, so this cannot be unit
tested in-process. Add an integration test that spawns the CLI as a child process
against a fixture tree, under the existing `test:integration` script.
## Risks
1. **`module.register()` is irreversible and process-wide.** It affects
everything loaded afterwards, including the CLI's own lazy imports. Guarded
single install, `next()`-first ordering.
2. **Hard-error duplicates have no escape hatch.** Two bundles claiming one id
cannot be used together, full stop. If that bites in practice the follow-up is
a `cubeAliases` map or a per-package id prefix — explicitly out of scope here.
3. **Store corruption.** A bundled cube writing to its own directory damages the
pnpm global store for every project on the machine. Documented in Phase 1;
a runtime warning is a possible follow-up.
4. **No version compatibility check.** A bundle authored against a future `nopy`
loaded by an older one fails at manifest-import time with a confusing error.
A `nopy.engines` field checked at resolution time would fix it. Deferred.
5. **Bundles vendoring cubes in their own `node_modules`** will not be found, by
design.
6. **`workspace:*` escaping into a published manifest.** The failure is silent at
publish time and only shows up when someone installs the package. Phase 4
fixes the lane; a `postpack` assertion that no dependency range starts with
`workspace:` would make it impossible to regress.
7. **Three packages, three version lines.** `nopy-cube` is the contract, so a
breaking change there ripples to every published bundle in the wild — which is
the point of versioning it separately, but it means the compatibility question
from risk 4 gets more pressing, not less.
+4 -8
View File
@@ -15,7 +15,7 @@ Hooks are defined as an array of functions in the cube manifest.
```javascript
import { z } from 'zod';
import { cubes } from '@bitstack/nopy';
import { cubes } from '@bitsquare/nopy';
export default cubes.Manifest({
name: 'my-cube',
@@ -51,19 +51,15 @@ Hooks can be synchronous or asynchronous (returning a `Promise`).
## Mechanics
### Sequential Execution
### Execution Order
In sequential execution mode (the default), cubes added via hooks will follow the order in which they were pushed to the deployment plan:
Cubes are always deployed sequentially, in the order they were pushed to the deployment plan. For a cube with hooks that means:
1. Cubes from `before` hooks.
2. The current cube itself.
3. Cubes from `after` hooks.
### Parallel Execution
In parallel execution mode, cubes added via hooks **do not automatically inherit dependencies**.
If a `before` hook calls `exec('setup-cube')`, it ensures that `setup-cube` is placed earlier in the deployment plan, but for parallel execution, you should still ensure that dependencies are correctly specified if one cube relies on another's completion.
Because a `before` hook only places its cube *earlier in the plan*, it guarantees ordering but not much else — if the relationship is a real dependency rather than a one-off ordering nudge, declare it in `dependencies` so it is resolved and deduplicated like any other.
### Variable Passing
+15
View File
@@ -7,10 +7,12 @@ This document tracks the major refactoring of the `nopy` package.
### 1. Remove parallel execution
- **Status**: ✅ Completed
- **Goal**: Remove all logic supporting parallel execution of cubes to simplify the execution flow and improve reliability.
- **Rationale**: The feature never shipped. Concurrent pyinfra processes interleave their output, which made deployment logs unreadable — a cost that outweighed the wall-clock saving. Do not reintroduce it without first solving per-cube output buffering.
- **Context**:
- Parallelism removed from `NopyConfig`, `NopyOptions`, and `executeDeployCalls`.
- `buildExecutionStages` deleted.
- CLI flags `--parallel` and `--concurrency` removed.
- Documentation caught up later: `README.md`, `docs/API.md`, and `docs/HOOKS.md` had all continued to describe the feature as if it existed.
- **Proposed Solution**: (Done)
### 2. Rework cube building process & Dependency Resolution
@@ -39,3 +41,16 @@ This document tracks the major refactoring of the `nopy` package.
- `Cube` is a class encapsulating a `Manifest` and runtime info (`dir`, `deployScript`).
- **Proposed Solution**: (Done)
### 5. Make `--use-defaults` operational
- **Status**: ✅ Completed
- **Goal**: Turn `-D` from a flag that was parsed and threaded through three layers but never read into a working non-interactive mode.
- **Rationale**: Unattended runs — CI, or provisioning a fresh box from a checked-in `.nopyrc.json` — are the reason the flag exists. It prompted anyway.
- **Context**:
- `BuildContext.resolveCube` branches on `options.useDefaults` and skips `VariableAssignment`.
- `Variables.get()` merge order corrected to defaults → global `env` → prompts → params. `env` used to lose to the schema default, which left a non-interactive run with no way to be configured at all.
- Replayed session values moved from the `defaults` scope to `prompts`, so they keep outranking `env` now that `env` sits higher.
- `Cube.getDefaults()` no longer discards every default when one field lacks `.default()`; it falls back to a per-field read.
- `Cube.requiredKeys()` added, and a `-D` run fails naming the unfillable variables instead of deploying a cube with them absent from `--data`.
- `VariableAssignment` offers every schema key, not only the ones carrying a default, and shows the value the run would actually use as the initial.
- **Proposed Solution**: (Done)
+1 -1
View File
@@ -252,7 +252,7 @@ generateSession({
Both formats are loaded the same way:
```javascript
import { loadSession } from '@bitstack/nopy';
import { loadSession } from '@bitsquare/nopy';
// Load JSON
const jsonSession = await loadSession('./my-session.session.json');
+2 -2
View File
@@ -1,6 +1,6 @@
{
"name": "@bitstack/nopy",
"version": "1.0.0-alpha4",
"name": "@bitsquare/nopy",
"version": "1.0.0-alpha5",
"description": "A system to simplify pyinfra script management and execution.",
"keywords": [
"pyinfra",
+27 -1
View File
@@ -32,11 +32,32 @@ export class BuildContext {
password?: string;
},
public readonly options: {
/** Skip the variable prompts and take whatever the non-interactive scopes hold. */
useDefaults?: boolean;
isSessionReplay?: boolean;
} = {}
) {}
/**
* Fails a non-interactive run that cannot fill a required variable.
*
* Without this the cube would be deployed with the key simply absent from
* `--data`, and the deploy script would read `None` off `host.data`.
*/
private assertVariablesComplete(cube: Cube): void {
const resolved = this.variables.get(cube.id);
const missing = cube.requiredKeys().filter((key) => resolved[key] === undefined);
if (missing.length === 0) return;
const [one, them] =
missing.length === 1 ? ['has no default value', 'it'] : ['have no default values', 'them'];
throw new Error(
`Cube "${cube.id}" cannot run with --use-defaults: ${missing.join(', ')} ${one}. ` +
`Set ${them} under "env" in .nopyrc.json, pass ${them} from a dependency, ` +
'or drop --use-defaults to be prompted.'
);
}
/**
* Resolves a cube, its dependencies, and hooks recursively
*/
@@ -60,10 +81,15 @@ export class BuildContext {
// 2. Variable collection
if (this.options.isSessionReplay) {
// Recorded answers go back into the scope they came from, so a replay
// reproduces them even when `env` sets the same key to something else.
const sessionCube = this.session.cubes.find((c) => c.key === cubeId);
if (sessionCube) {
this.variables.assign(cubeId, 'defaults', sessionCube.variables);
this.variables.assign(cubeId, 'prompts', sessionCube.variables);
}
} else if (this.options.useDefaults) {
log.debug('Skipping prompts, using defaults', { cubeId });
this.assertVariablesComplete(cube);
} else {
await VariableAssignment(cube, this.variables);
}
+2
View File
@@ -31,6 +31,8 @@ export type {
export {
Cube,
Manifest,
zodInner,
zodKind,
} from './types.js';
// Utilities
+88 -25
View File
@@ -50,16 +50,31 @@ function extractCubeId(manifest: Manifest): string | undefined {
return match ? match[1] : undefined;
}
/**
* Loads all cubes from discovered cube directories.
*/
export async function loadCubes(): Promise<LoadResult> {
const cubesFolders = findCubeDirectories();
const cubes: Record<string, Cube> = {};
const errors: string[] = [];
/** A cube found on disk, before ids have been checked against each other. */
interface CubeCandidate {
id: string;
manifest: Manifest;
dir: string;
deployScript: string;
}
async function scanDirectory(currentDir: string, baseDir: string): Promise<void> {
const entries = await fs.readdir(currentDir, { withFileTypes: true });
/** What one root directory contributed. */
interface ScanResult {
candidates: CubeCandidate[];
errors: string[];
}
/**
* Walks one root, collecting every cube below it.
*
* Deliberately does not decide anything about ids: a duplicate is only visible
* once every root has been walked, and stopping the descent here would hide
* whatever sits below the offending directory.
*/
async function scanDirectory(currentDir: string, result: ScanResult): Promise<void> {
const entries = (await fs.readdir(currentDir, { withFileTypes: true })).sort((a, b) =>
a.name.localeCompare(b.name)
);
const files = entries.filter((e) => e.isFile());
const manifestFile = files.find(
@@ -68,50 +83,98 @@ export async function loadCubes(): Promise<LoadResult> {
const deployFile = files.find((f) => f.name === 'deploy.py' || f.name.endsWith('.deploy.py'));
if (manifestFile && deployFile) {
const cubePath = currentDir;
const manifestPath = path.join(cubePath, manifestFile.name);
const manifestPath = path.join(currentDir, manifestFile.name);
try {
const manifest = (await import(manifestPath)).default as Manifest;
if (!manifest || typeof manifest !== 'object') {
errors.push(`Invalid manifest export in ${manifestPath}`);
result.errors.push(`Invalid manifest export in ${manifestPath}`);
} else if (!manifest.name) {
errors.push(`Invalid manifest format in ${manifestPath}: missing 'name'`);
result.errors.push(`Invalid manifest format in ${manifestPath}: missing 'name'`);
} else {
const cubeId = extractCubeId(manifest) || path.basename(cubePath);
if (cubes[cubeId]) {
errors.push(`Duplicate cube id '${cubeId}'`);
return;
}
const cubeId = extractCubeId(manifest) || path.basename(currentDir);
// Ensure basic properties
manifest.id = cubeId;
manifest.schema = manifest.schema ?? z.object({});
cubes[cubeId] = new Cube(manifest, cubePath, deployFile.name);
result.candidates.push({
id: cubeId,
manifest,
dir: currentDir,
deployScript: deployFile.name,
});
}
} catch (err) {
errors.push(`Failed to load manifest ${manifestPath}: ${err}`);
result.errors.push(`Failed to load manifest ${manifestPath}: ${err}`);
}
}
for (const entry of entries) {
if (entry.isDirectory() && !entry.name.startsWith('.') && entry.name !== 'node_modules') {
await scanDirectory(path.join(currentDir, entry.name), baseDir);
}
await scanDirectory(path.join(currentDir, entry.name), result);
}
}
}
await Promise.all(
/** The message a duplicate id produces. Aborts the run — see `nopy.main.ts`. */
function duplicateError(id: string, group: CubeCandidate[]): string {
const where = group.map((c) => ` ${c.dir}`).join('\n');
return (
`Duplicate cube id '${id}' from ${group.length} sources:\n${where}\n` +
`Rename one of them, or remove a source from .nopyrc.json.`
);
}
/**
* Loads all cubes from discovered cube directories.
*
* Scanning and id resolution are separate passes on purpose. Each root
* contributes its own candidate list, and those lists are concatenated in
* root order rather than in whichever order the concurrent scans happened to
* finish — so which cube is reported as "the duplicate" is the same on every
* run, which is what makes the hard error testable.
*/
export async function loadCubes(): Promise<LoadResult> {
const cubesFolders = findCubeDirectories();
const scans = await Promise.all(
cubesFolders.map(async (folder) => {
const result: ScanResult = { candidates: [], errors: [] };
if (fs.existsSync(folder)) {
await scanDirectory(folder, folder);
await scanDirectory(folder, result);
}
return result;
})
);
// Promise.all preserves input order regardless of completion order.
const errors = scans.flatMap((scan) => scan.errors);
// One directory reachable from two roots (a `cubeDirs` entry nested under a
// `.npcubes` marker, say) is one cube seen twice, not a collision.
const seenDirs = new Set<string>();
const byId = new Map<string, CubeCandidate[]>();
for (const candidate of scans.flatMap((scan) => scan.candidates)) {
if (seenDirs.has(candidate.dir)) continue;
seenDirs.add(candidate.dir);
const group = byId.get(candidate.id);
if (group) group.push(candidate);
else byId.set(candidate.id, [candidate]);
}
const cubes: Record<string, Cube> = {};
for (const [id, group] of byId) {
if (group.length > 1) errors.push(duplicateError(id, group));
// The map is still populated for the callers that only report; a duplicate
// is fatal, so which candidate landed here never reaches a deploy.
const [first] = group;
cubes[id] = new Cube(first.manifest, first.dir, first.deployScript);
}
return { cubes, errors };
}
+85 -6
View File
@@ -82,6 +82,62 @@ export namespace Manifest {
}
}
/**
* zod's runtime discriminant for a schema node, as a plain string.
*
* `instanceof z.ZodDefault` compares against the *running* copy of zod. A cube
* manifest is free to build its schema with a different copy — its own
* dependency, or one shipped inside a bundle — and then every `instanceof`
* quietly returns false and the caller falls through to a wrong answer instead
* of failing. `def.type` holds across instances, so nothing here may go back to
* `instanceof`.
*/
export function zodKind(zodType: unknown): string {
return (zodType as { def: { type: string } }).def.type;
}
/**
* The type a wrapper wraps — `.default()`, `.optional()`, `.nullable()`.
* Only call this for a node whose {@link zodKind} is one of those.
*/
export function zodInner(zodType: unknown): z.ZodType {
return (zodType as { def: { innerType: z.ZodType } }).def.innerType;
}
/**
* Reads the `.default()` off a schema field, unwrapping the wrappers that may
* sit above it (`.default().optional()`, `.default().nullable()`).
*
* Returns `undefined` for a field that declares no default — which is also how
* `requiredKeys()` recognises a field the user has to supply.
*/
function defaultValueOf(zodType: z.ZodType): unknown {
const kind = zodKind(zodType);
if (kind === 'default') {
// zod 4 exposes `defaultValue` as a getter that already invokes a lazily
// declared default; the function branch is insurance against that changing.
const { defaultValue } = (zodType as unknown as { def: { defaultValue: unknown } }).def;
return typeof defaultValue === 'function' ? defaultValue() : defaultValue;
}
if (kind === 'optional' || kind === 'nullable') {
return defaultValueOf(zodInner(zodType));
}
return undefined;
}
/**
* Where a cube was discovered.
*
* Worth carrying because a cube's own directory does not say how it got into
* the run: `/…/node_modules/@acme/cubes-net/cubes/x` could equally have come
* from a `cubeDirs` entry pointing straight at it.
*/
export type CubeSource =
/** Found under a `cubeDirs` entry or a `.npcubes` marker, at `dir`. */
| { type: 'dir'; dir: string }
/** Contributed by a package named in `cubePackages`. */
| { type: 'package'; packageName: string; dir: string };
/**
* A fully loaded cube with its filesystem location and runtime state
*/
@@ -89,7 +145,9 @@ export class Cube<Schema extends AnyObjectSchema = AnyObjectSchema> {
constructor(
public readonly manifest: Manifest<Schema>,
public readonly dir: string,
public readonly deployScript: string
public readonly deployScript: string,
/** Defaults to the cube's own directory, for cubes built by hand. */
public readonly source: CubeSource = { type: 'dir', dir }
) {}
get id(): string {
@@ -101,14 +159,35 @@ export class Cube<Schema extends AnyObjectSchema = AnyObjectSchema> {
}
/**
* Returns default values for the cube's schema
* Returns default values for the cube's schema.
*
* Parsing an empty object resolves every default in one go, but it fails
* outright as soon as one field has no `.default()`. Falling back to a
* per-field read keeps the defaults that *are* declared instead of dropping
* the whole set — a single required field used to leave the cube with no
* variables at all.
*/
getDefaults(): z.infer<Schema> {
try {
return this.manifest.schema.parse({});
} catch {
return {} as z.infer<Schema>;
const parsed = this.manifest.schema.safeParse({});
if (parsed.success) return parsed.data as z.infer<Schema>;
const defaults: Record<string, unknown> = {};
for (const [key, zodType] of Object.entries(this.manifest.schema.shape)) {
const value = defaultValueOf(zodType);
if (value !== undefined) defaults[key] = value;
}
return defaults as z.infer<Schema>;
}
/**
* Schema keys that have to be supplied from somewhere: no `.default()`, and
* not optional. Nothing else can fill them in, so a run that cannot prompt
* has to fail rather than deploy a cube with the value missing.
*/
requiredKeys(): string[] {
return Object.entries(this.manifest.schema.shape)
.filter(([, zodType]) => !zodType.safeParse(undefined).success)
.map(([key]) => key);
}
}
+10 -2
View File
@@ -19,7 +19,6 @@ export class Variables {
constructor(readonly global: TVariables = {}) {}
assign(artefactId: Variables.ArtefactId, scope: Variables.Scope, values: TVariables = {}) {
console.log('Assigning', artefactId, scope, values);
if (!this[scope][artefactId]) {
this[scope][artefactId] = values;
} else {
@@ -27,13 +26,22 @@ export class Variables {
}
}
/**
* Merges the scopes for one cube, lowest precedence first:
* schema defaults → global `env` → prompts (or replayed session values) →
* params handed over by a dependency or a hook.
*
* Defaults sit at the bottom so `env` in `.nopyrc.json` can steer a run that
* never prompts (`--use-defaults`); a key that a dependency supplies is never
* prompted for, so prompts and params do not compete in practice.
*/
get(artefactId: Variables.ArtefactId, scope?: Variables.Scope): TVariables {
if (scope) {
return this[scope][artefactId] || {};
}
return {
...this.global,
...this.defaults[artefactId],
...this.global,
...this.prompts[artefactId],
...this.params[artefactId],
};
+42 -8
View File
@@ -55,13 +55,41 @@ export type ResolutionConfig = {
};
/**
* Raw config file structure (includes resolution)
* A cube package named in `cubePackages`, paired with where it was named.
*
* Node resolution has to start from the config file that asked for the package,
* not from `process.cwd()` — otherwise a package listed in `~/.nopyrc.json`
* only resolves in projects that happen to depend on it themselves. This is the
* same problem {@link PATH_PROPERTIES} solves for `cubeDirs`, except the answer
* is a reference to resolve later rather than a rewritten path.
*/
export interface NopyConfigFile extends Partial<NopyConfig> {
export interface CubePackageRef {
/** The package name as written in the config, e.g. `@acme/cubes-net`. */
spec: string;
/** Directory of the `.nopyrc.json` that named it. */
from: string;
}
/**
* Raw config file structure (includes resolution)
*
* Diverges from {@link NopyConfig} for `cubePackages`: a file lists plain
* package names, and loading turns each into a {@link CubePackageRef}.
*/
export interface NopyConfigFile extends Omit<Partial<NopyConfig>, 'cubePackages'> {
/** Cube packages to load, by package name */
cubePackages?: string[];
/** Customize merge behavior for specific properties */
resolution?: ResolutionConfig;
}
/**
* A config file whose paths have been resolved — what actually gets merged.
*/
type ResolvedConfigFile = Omit<NopyConfigFile, 'cubePackages'> & {
cubePackages?: CubePackageRef[];
};
/**
* Nopy configuration file structure
*/
@@ -70,6 +98,8 @@ export interface NopyConfig {
hosts: string[];
/** Directories to search for cubes */
cubeDirs: string[];
/** Installed packages to load cubes from */
cubePackages: CubePackageRef[];
/** Global environment variables */
env: TVariables;
/** Logging configuration */
@@ -86,6 +116,7 @@ export interface NopyConfig {
const DEFAULT_CONFIG: NopyConfig = {
hosts: [],
cubeDirs: [],
cubePackages: [],
env: {},
};
@@ -219,30 +250,33 @@ const PATH_PROPERTIES: (keyof NopyConfig)[] = ['cubeDirs'];
* Resolves relative paths in a config file based on its location
* Only resolves paths for properties that are known to contain filesystem paths
*/
function resolveConfigPaths(config: NopyConfigFile, configPath: string): NopyConfigFile {
function resolveConfigPaths(config: NopyConfigFile, configPath: string): ResolvedConfigFile {
const configDir = path.dirname(configPath);
const resolved: NopyConfigFile = {};
const resolved: Record<string, unknown> = {};
for (const [key, value] of Object.entries(config)) {
if (key === 'resolution') {
// Don't resolve the resolution config itself
resolved[key] = value as ResolutionConfig;
} else if (key === 'cubePackages') {
// Not a path — a package name, tagged with where to resolve it from.
resolved[key] = (value as string[]).map((spec) => ({ spec, from: configDir }));
} else if (PATH_PROPERTIES.includes(key as keyof NopyConfig)) {
// Only resolve paths for known path properties
resolved[key as keyof NopyConfigFile] = resolveRelativePaths(value, configDir) as any;
resolved[key] = resolveRelativePaths(value, configDir);
} else {
// Copy other properties as-is (including hosts)
resolved[key as keyof NopyConfigFile] = value as any;
resolved[key] = value;
}
}
return resolved;
return resolved as ResolvedConfigFile;
}
/**
* Merges a child config into a parent config
*/
function mergeConfigs(parent: NopyConfig, childFile: NopyConfigFile): NopyConfig {
function mergeConfigs(parent: NopyConfig, childFile: ResolvedConfigFile): NopyConfig {
const resolution = childFile.resolution || {};
const result: Record<string, unknown> = { ...parent };
+32 -14
View File
@@ -6,8 +6,8 @@
import Enquirer from 'enquirer';
import fuzzy from 'fuzzy';
import inquirer from 'inquirer';
import { z } from 'zod';
import type { AnyObjectSchema, Cube } from './cubes/index.js';
import type { z } from 'zod';
import { type AnyObjectSchema, type Cube, zodInner, zodKind } from './cubes/index.js';
import type { Variables } from './nopy.common.js';
interface CubeChoice {
@@ -142,20 +142,32 @@ export async function HostSelection(hosts: string[]): Promise<string> {
return selectedHost.customHost ?? selectedHost.host;
}
/**
* Turns a form answer — always a string — back into what the schema declares.
*
* Discriminates on {@link zodKind} rather than `instanceof`: the schema may
* have been built by a copy of zod that is not the one this file imported, and
* `instanceof` would then fail open and leave every value a string.
*/
function coerceValue(value: unknown, zodType: z.core.$ZodType): unknown {
if (typeof value !== 'string') return value;
if (zodType instanceof z.ZodDefault) return coerceValue(value, zodType._def.innerType);
if (zodType instanceof z.ZodOptional) return coerceValue(value, zodType._def.innerType);
if (zodType instanceof z.ZodNullable) {
switch (zodKind(zodType)) {
case 'default':
case 'optional':
return coerceValue(value, zodInner(zodType));
case 'nullable':
if (value === 'null' || value === '') return null;
return coerceValue(value, zodType._def.innerType);
}
if (zodType instanceof z.ZodBoolean) return value === 'true' || value === 'yes' || value === '1';
if (zodType instanceof z.ZodNumber) {
return coerceValue(value, zodInner(zodType));
case 'boolean':
return value === 'true' || value === 'yes' || value === '1';
case 'number': {
const num = Number(value);
return Number.isNaN(num) ? value : num;
}
default:
return value;
}
}
interface FormChoice {
@@ -169,13 +181,19 @@ export async function VariableAssignment<S extends AnyObjectSchema>(
variables: Variables
) {
const schema = cube.manifest.schema.shape;
const defaults = cube.getDefaults();
const defaults = cube.getDefaults() as Record<string, unknown>;
const params = variables.get(cube.id, 'params');
const resolved = variables.get(cube.id);
const variablesToConfigure: Record<string, unknown> = {};
for (const [key, defaultValue] of Object.entries(defaults)) {
if (variables.get(cube.id, 'params')[key] === undefined) {
variablesToConfigure[key] = defaultValue;
}
// Every schema key is offered, not just the ones carrying a `.default()` — a
// field without one is precisely the field that has to be asked about. Keys a
// dependency or hook already supplied are left alone. The value shown is the
// one the run would otherwise use, so `env` from `.nopyrc.json` is visible
// (and editable) rather than silently overridden by whatever is typed.
for (const key of Object.keys(schema)) {
if (params[key] !== undefined) continue;
variablesToConfigure[key] = resolved[key] ?? defaults[key];
}
if (Object.keys(variablesToConfigure).length === 0) return;
+78
View File
@@ -0,0 +1,78 @@
/**
* Tests for the Variables scope container.
*
* The merge order is what makes a non-interactive run configurable, so it is
* pinned down here rather than left to the callers to demonstrate.
*/
import { describe, expect, it } from 'vitest';
import { Variables } from '../src/nopy.common.js';
describe('Variables.assign', () => {
it('creates the scope entry on first assignment and merges afterwards', () => {
const variables = new Variables();
variables.assign('cube-a', 'defaults', { A: 1 });
variables.assign('cube-a', 'defaults', { B: 2 });
expect(variables.get('cube-a', 'defaults')).toEqual({ A: 1, B: 2 });
});
it('defaults to an empty assignment', () => {
const variables = new Variables();
variables.assign('cube-a', 'prompts');
expect(variables.get('cube-a', 'prompts')).toEqual({});
});
it('keeps scopes and cubes apart', () => {
const variables = new Variables();
variables.assign('cube-a', 'params', { A: 1 });
expect(variables.get('cube-a', 'prompts')).toEqual({});
expect(variables.get('cube-b', 'params')).toEqual({});
});
});
describe('Variables.get precedence', () => {
it('lets global env override a schema default', () => {
const variables = new Variables({ PORT: 2222 });
variables.assign('cube-a', 'defaults', { PORT: 22 });
expect(variables.get('cube-a').PORT).toBe(2222);
});
it('lets a prompt override global env', () => {
const variables = new Variables({ PORT: 2222 });
variables.assign('cube-a', 'defaults', { PORT: 22 });
variables.assign('cube-a', 'prompts', { PORT: 8080 });
expect(variables.get('cube-a').PORT).toBe(8080);
});
it('lets a dependency param override everything else', () => {
const variables = new Variables({ PORT: 2222 });
variables.assign('cube-a', 'defaults', { PORT: 22 });
variables.assign('cube-a', 'prompts', { PORT: 8080 });
variables.assign('cube-a', 'params', { PORT: 9090 });
expect(variables.get('cube-a').PORT).toBe(9090);
});
it('merges keys from every scope', () => {
const variables = new Variables({ G: 'g' });
variables.assign('cube-a', 'defaults', { D: 'd' });
variables.assign('cube-a', 'prompts', { P: 'p' });
variables.assign('cube-a', 'params', { X: 'x' });
expect(variables.get('cube-a')).toEqual({ G: 'g', D: 'd', P: 'p', X: 'x' });
});
it('applies global env to every cube', () => {
const variables = new Variables({ SHARED: 'yes' });
expect(variables.get('anything').SHARED).toBe('yes');
});
});
@@ -104,6 +104,99 @@ describe('BuildContext session replay', () => {
});
});
describe('BuildContext --use-defaults', () => {
const withDefaults = (cube: Cube, variables = new Variables(), cfg = config) =>
new BuildContext(
{ [cube.id]: cube },
variables,
session(),
cfg,
{ method: 'ssh' },
{ useDefaults: true }
);
it('skips the prompts and deploys the schema defaults', async () => {
const cube = testCube('cube-a', z.object({ PORT: z.string().default('3000') }));
const context = withDefaults(cube);
await context.resolveCube('cube-a', 'host1');
expect(VariableAssignment).not.toHaveBeenCalled();
expect(context.deployCalls[0].env.PORT).toBe('3000');
});
it('lets global env steer the run', async () => {
const cube = testCube('cube-a', z.object({ PORT: z.string().default('3000') }));
const context = withDefaults(cube, new Variables({ PORT: '8080' }));
await context.resolveCube('cube-a', 'host1');
expect(context.deployCalls[0].command.join(' ')).toContain('--data "PORT=8080"');
});
it('refuses to run a cube whose variable nothing can supply', async () => {
const cube = testCube('cube-a', z.object({ SSID: z.string(), PSK: z.string() }));
const context = withDefaults(cube);
await expect(context.resolveCube('cube-a', 'host1')).rejects.toThrow(
/Cube "cube-a" cannot run with --use-defaults: SSID, PSK have no default values/
);
expect(context.deployCalls).toHaveLength(0);
});
it('names a single missing variable in the singular', async () => {
const cube = testCube('cube-a', z.object({ SSID: z.string() }));
await expect(withDefaults(cube).resolveCube('cube-a', 'host1')).rejects.toThrow(
'SSID has no default value'
);
});
it('accepts a required variable supplied by global env', async () => {
const cube = testCube('cube-a', z.object({ SSID: z.string() }));
const context = withDefaults(cube, new Variables({ SSID: 'home' }));
await context.resolveCube('cube-a', 'host1');
expect(context.deployCalls[0].env.SSID).toBe('home');
});
it('accepts a required variable supplied by a dependency', async () => {
const cube = testCube('cube-a', z.object({ SSID: z.string() }));
const context = withDefaults(cube);
await context.resolveCube('cube-a', 'host1', { SSID: 'from-dep' });
expect(context.deployCalls[0].env.SSID).toBe('from-dep');
});
it('still resolves dependencies and hooks', async () => {
const dep = testCube('dep');
const main = new Cube(
Manifest.create({
id: 'main',
name: 'Main',
schema: z.object({ FLAG: z.boolean().default(true) }),
dependencies: (vars: Record<string, unknown>) => (vars.FLAG ? ['dep'] : []),
}),
'/test/main',
'deploy.py'
);
const context = new BuildContext(
{ dep, main },
new Variables(),
session(),
config,
{ method: 'ssh' },
{ useDefaults: true }
);
await context.resolveCube('main', 'host1');
expect(context.deployCalls.map((c) => c.cube)).toEqual(['dep', 'main']);
});
});
describe('BuildContext command construction', () => {
const build = (auth: { method: string; username?: string; password?: string }) => {
const context = new BuildContext(
+51 -2
View File
@@ -136,14 +136,63 @@ describe('loader edge cases', () => {
expect(errors[0]).toMatch(/Failed to load manifest/);
});
it('reports duplicate cube ids', async () => {
it('reports duplicate cube ids, naming every directory that claims one', async () => {
cube('first', 'export default { id: "dup", name: "First" }');
cube('second', 'export default { id: "dup", name: "Second" }');
const { cubes, errors } = await loadCubes();
expect(Object.keys(cubes)).toEqual(['dup']);
expect(errors[0]).toMatch(/Duplicate cube id 'dup'/);
expect(errors).toHaveLength(1);
expect(errors[0]).toMatch(/Duplicate cube id 'dup' from 2 sources/);
expect(errors[0]).toContain(path.join(tmpDir, 'first'));
expect(errors[0]).toContain(path.join(tmpDir, 'second'));
});
it('keeps scanning below a duplicate instead of dropping the subtree', async () => {
cube('first', 'export default { id: "dup", name: "First" }');
cube('second', 'export default { id: "dup", name: "Second" }');
cube('second/inner', 'export default { id: "buried", name: "Buried" }');
const { cubes, errors } = await loadCubes();
expect(cubes.buried).toBeDefined();
expect(errors).toHaveLength(1);
});
it('reports the same duplicate whichever root is scanned first', async () => {
cube('a/one', 'export default { id: "dup", name: "One" }');
cube('b/two', 'export default { id: "dup", name: "Two" }');
const roots = [path.join(tmpDir, 'a'), path.join(tmpDir, 'b')];
fs.writeFileSync(path.join(tmpDir, '.nopyrc.json'), JSON.stringify({ cubeDirs: roots }));
const forwards = await loadCubes();
fs.writeFileSync(
path.join(tmpDir, '.nopyrc.json'),
JSON.stringify({ cubeDirs: [...roots].reverse() })
);
const backwards = await loadCubes();
expect(forwards.errors[0]).toContain(path.join(tmpDir, 'a', 'one'));
expect(forwards.errors[0]).toContain(path.join(tmpDir, 'b', 'two'));
expect(backwards.errors).toHaveLength(1);
expect(new Set(backwards.errors[0].split('\n'))).toEqual(
new Set(forwards.errors[0].split('\n'))
);
});
it('does not call one directory a duplicate of itself when two roots reach it', async () => {
cube('nested/one', 'export default { id: "once", name: "Once" }');
fs.writeFileSync(
path.join(tmpDir, '.nopyrc.json'),
JSON.stringify({ cubeDirs: ['./', './nested'] })
);
const { cubes, errors } = await loadCubes();
expect(errors).toEqual([]);
expect(cubes.once).toBeDefined();
});
it('skips hidden and node_modules directories', async () => {
+111
View File
@@ -0,0 +1,111 @@
/**
* Tests for the Cube runtime wrapper: default extraction and the required-key
* check that `--use-defaults` relies on.
*/
import { describe, expect, it } from 'vitest';
import { z } from 'zod';
import { Cube, Manifest } from '../src/cubes/types.js';
import { foreignZodSchema } from './helpers/foreign-zod.js';
const cube = (schema: z.ZodObject<any>) =>
new Cube(Manifest.create({ id: 'c', name: 'C', schema }), '/cubes/c', 'deploy.py');
describe('Cube.getDefaults', () => {
it('resolves every default when the whole schema parses', () => {
const c = cube(
z.object({
PORT: z.number().default(8080),
NAME: z.string().default('svc'),
})
);
expect(c.getDefaults()).toEqual({ PORT: 8080, NAME: 'svc' });
});
it('keeps the declared defaults when one field has none', () => {
const c = cube(
z.object({
SSID: z.string(),
PRIORITY: z.number().default(10),
HIDDEN: z.boolean().default(false),
})
);
expect(c.getDefaults()).toEqual({ PRIORITY: 10, HIDDEN: false });
});
it('unwraps a default sitting under optional or nullable', () => {
const c = cube(
z.object({
REQUIRED: z.string(),
A: z.number().default(1).optional(),
B: z.number().default(2).nullable(),
C: z.number().optional().default(3),
})
);
expect(c.getDefaults()).toEqual({ A: 1, B: 2, C: 3 });
});
it('evaluates a lazily declared default', () => {
const c = cube(
z.object({ REQUIRED: z.string(), TOKEN: z.string().default(() => 'generated') })
);
expect(c.getDefaults()).toEqual({ TOKEN: 'generated' });
});
it('omits an optional field that declares no default', () => {
const c = cube(z.object({ REQUIRED: z.string(), MAYBE: z.string().optional() }));
expect(c.getDefaults()).toEqual({});
});
it('returns an empty object for an empty schema', () => {
expect(cube(z.object({})).getDefaults()).toEqual({});
});
it('reads defaults off a schema built by a different copy of zod', () => {
// The per-field fallback reads zod's internals directly. Under `instanceof`
// a foreign schema yields no defaults at all, without erroring.
const c = cube(
foreignZodSchema(
z.object({
REQUIRED: z.string(),
PRIORITY: z.number().default(10),
NESTED: z.number().default(2).optional(),
})
)
);
expect(c.getDefaults()).toEqual({ PRIORITY: 10, NESTED: 2 });
});
});
describe('Cube.requiredKeys', () => {
it('lists the fields with neither a default nor optionality', () => {
const c = cube(
z.object({
SSID: z.string(),
PASSWORD: z.string(),
PRIORITY: z.number().default(10),
NOTE: z.string().optional(),
})
);
expect(c.requiredKeys()).toEqual(['SSID', 'PASSWORD']);
});
it('treats a nullable field without a default as required', () => {
const c = cube(z.object({ MAYBE: z.string().nullable() }));
expect(c.requiredKeys()).toEqual(['MAYBE']);
});
it('is empty when every field can fill itself in', () => {
const c = cube(z.object({ A: z.string().default('a'), B: z.string().optional() }));
expect(c.requiredKeys()).toEqual([]);
});
});
@@ -0,0 +1,29 @@
/**
* A schema that behaves like zod's but does not share zod's prototypes.
*
* Once cubes arrive from `node_modules`, the schema a manifest builds may come
* from a *second* copy of zod — its own dependency, or one shipped inside a
* bundle. Such a schema is structurally identical and `instanceof` blind to it.
* Rebuilding the nodes as plain objects reproduces that from inside a single
* process, so anything that reads zod's internals stays pinned to `def.type`.
*/
import type { z } from 'zod';
/** Strips the prototype off a schema node and everything it wraps. */
function strip(node: unknown): unknown {
const def = { ...(node as { def: Record<string, unknown> }).def };
if (def.innerType) def.innerType = strip(def.innerType);
return { def };
}
export function foreignZodSchema<S extends z.ZodObject<any>>(schema: S): S {
return {
// Parsing is not what is under test — delegate it and keep the real
// behaviour, so only the introspection path sees the foreign nodes.
safeParse: (value: unknown) => schema.safeParse(value),
shape: Object.fromEntries(
Object.entries(schema.shape).map(([key, node]) => [key, strip(node)])
),
} as unknown as S;
}
+63 -3
View File
@@ -9,9 +9,10 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { z } from 'zod';
const { inquirerPrompt, formRun, autoCompleteRun, autoCompleteCtor } = vi.hoisted(() => ({
const { inquirerPrompt, formRun, formCtor, autoCompleteRun, autoCompleteCtor } = vi.hoisted(() => ({
inquirerPrompt: vi.fn(),
formRun: vi.fn(),
formCtor: vi.fn(),
autoCompleteRun: vi.fn(),
autoCompleteCtor: vi.fn(),
}));
@@ -23,6 +24,9 @@ vi.mock('enquirer', () => ({
default: {
Form: class {
run = formRun;
constructor(options: unknown) {
formCtor(options);
}
},
AutoComplete: class {
run = autoCompleteRun;
@@ -42,6 +46,7 @@ import {
PasswordSelection,
VariableAssignment,
} from '../src/nopy.prompts.js';
import { foreignZodSchema } from './helpers/foreign-zod.js';
/** Grabs the single question object passed to the last inquirer.prompt call. */
const questions = () => inquirerPrompt.mock.calls.at(-1)?.[0] as Record<string, any>[];
@@ -50,6 +55,12 @@ const question = (name: string) => questions().find((q) => q.name === name);
/** Grabs the options the last enquirer AutoComplete prompt was constructed with. */
const autoComplete = () => autoCompleteCtor.mock.calls.at(-1)?.[0] as Record<string, any>;
/** Grabs the choices the last enquirer Form prompt was constructed with. */
const formChoices = () => {
const options = formCtor.mock.calls.at(-1)?.[0] as { choices: Record<string, any>[] };
return options.choices;
};
const cube = (id: string, name: string, schema = z.object({})) =>
new Cube(Manifest({ id, name, schema }), `/cubes/${id}`, 'deploy.py');
@@ -240,6 +251,30 @@ describe('VariableAssignment', () => {
expect(variables.get('svc', 'prompts')).toEqual({});
});
it('asks about a field that declares no default, with an empty initial value', async () => {
const required = z.object({
SSID: z.string().describe('Network name'),
PRIORITY: z.number().default(10),
});
formRun.mockResolvedValue({});
await VariableAssignment(cube('wifi', 'WiFi', required), new Variables());
expect(formChoices()).toEqual([
{ name: 'SSID', message: 'Network name', initial: '' },
{ name: 'PRIORITY', message: 'PRIORITY', initial: '10' },
]);
});
it('offers the value the run would use, not the bare schema default', async () => {
const variables = new Variables({ port: 2222 });
formRun.mockResolvedValue({});
await VariableAssignment(cube('svc', 'Service', schema), variables);
expect(formChoices().find((c) => c.name === 'port')?.initial).toBe('2222');
});
it('coerces answers using the schema and stores them under prompts', async () => {
const variables = new Variables();
formRun.mockResolvedValue({ port: '9090', enabled: 'true', name: 'api' });
@@ -276,13 +311,14 @@ describe('VariableAssignment', () => {
const nullableSchema = z.object({
maybe: z.number().nullable().default(1),
opt: z.number().optional().default(2),
given: z.number().nullable().default(3),
});
const variables = new Variables();
formRun.mockResolvedValue({ maybe: 'null', opt: '7' });
formRun.mockResolvedValue({ maybe: 'null', opt: '7', given: '42' });
await VariableAssignment(cube('svc', 'Service', nullableSchema), variables);
expect(variables.get('svc', 'prompts')).toEqual({ maybe: null, opt: 7 });
expect(variables.get('svc', 'prompts')).toEqual({ maybe: null, opt: 7, given: 42 });
});
it('treats an empty string as null for a nullable field', async () => {
@@ -322,4 +358,28 @@ describe('VariableAssignment', () => {
).resolves.toBeUndefined();
expect(variables.get('svc', 'prompts')).toEqual({});
});
it('coerces against a schema built by a different copy of zod', async () => {
// Guards the discriminant in `coerceValue`: under `instanceof` every check
// here returns false and the answers stay strings, silently.
const variables = new Variables();
formRun.mockResolvedValue({ port: '9090', enabled: 'true', maybe: '' });
await VariableAssignment(
cube(
'svc',
'Service',
foreignZodSchema(
z.object({
port: z.number().default(8080),
enabled: z.boolean().default(false),
maybe: z.number().nullable().default(1),
})
)
),
variables
);
expect(variables.get('svc', 'prompts')).toEqual({ port: 9090, enabled: true, maybe: null });
});
});
+2 -2
View File
@@ -12,8 +12,8 @@
"skipLibCheck": true,
"resolveJsonModule": true,
"paths": {
"@bitstack/nopy": ["./packages/nopy/src"],
"@bitstack/keyman": ["./packages/keyman/src"]
"@bitsquare/nopy": ["./packages/nopy/src"],
"@bitsquare/keyman": ["./packages/keyman/src"]
}
},
"exclude": ["coverage", "node_modules", "dist"]