[docs] nopy: record the seven findings this branch closed
The documentation half of the same work: `DOCS-AUDIT.md` marks §1.3, §1.5, §2.3, §4.2 (all three points), §5.1, §6.1, §6.2 and §6.5 closed, each keeping its original text as the record with what closed it quoted underneath, and the "suggested order of attack" is rewritten to what is actually left — §5.2, §5.3, the two missing cube READMEs, and the two findings (§2.7, §4.4) that are stated accurately in `docs/API.md` while the code still behaves as they describe. `docs/API.md` drops the two entries from its *Known gaps* list that are no longer gaps, documents the argv and the absent shell, describes the resolution stack and the error it raises, and inverts the `.default()`/`.describe()` warning: the order used to matter and no longer does, which is worth saying outright since the old advice is in the reader's memory and in 15 manifests. The README's "topological sorting" becomes "in dependency order, with cycle detection" — the sort never existed, but until this branch neither did the thing a sort would have been for — and `--no-history` is spelled `--no-save-history` wherever it appears. One line of code rides along, because it is what a `docs/API.md` note has been asking for: `CubePackageRef` is re-exported from `src/index.ts`, so importing `NopyConfig` from `@bitsquare/nopy` no longer gives you a type whose own members you cannot name. The note in `docs/API.md` saying it is missing goes with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCzYTAm9QUhvLNr2EpdagJ
This commit is contained in:
Benjamin Diedrichsen
co-authored by
Claude Opus 5
Claude Opus 5
parent
b5702e423a
commit
05f2d6aa56
+16
-17
@@ -1,28 +1,30 @@
|
||||
# Nopy
|
||||
|
||||
A CLI tool that simplifies **pyinfra** script management and execution, providing an interactive workflow for deploying infrastructure configurations ("cubes") to remote hosts.
|
||||
A CLI tool that simplifies **pyinfra** script management and execution, providing an interactive workflow for deploying infrastructure configurations `cubes` to remote hosts.
|
||||
|
||||
## Overview
|
||||
|
||||
Nopy wraps pyinfra with structure, validation, and an interactive experience for managing complex infrastructure deployments. It organizes deployments into self-contained "cubes" with dependency management, schema validation, and lifecycle hooks.
|
||||
Nopy wraps [pyinfra](https://pyinfra.com/) in the javascript ecosystem to provide an interactive experience for managing repeatable infrastructure deployments. It organizes deployments into self-contained units - called `cubes` - adding support for transitive dependency management, user input validation, and different lifecycle hooks.
|
||||
|
||||
## Features
|
||||
## Features in a Nutshell
|
||||
|
||||
- **Dependency resolution** with topological sorting
|
||||
- **Before/after hooks** for multi-cube orchestration
|
||||
- **Manifest files** to support declarative description of user inputs and orchestration semantics per cube
|
||||
- **Dependency resolution** in dependency order, with cycle detection
|
||||
- **Before/after hooks** for programmable, multi-cube orchestration
|
||||
- **SSH key or password authentication**
|
||||
- **Default values** with optional customization via manifest `env`
|
||||
- **Schema validation** using Zod
|
||||
- **Schema validation** and **type coercion** using Zod
|
||||
- **Recursive cube directory discovery**
|
||||
- **Dry-run mode** for previewing deployments
|
||||
- **Dry-run mode** for previewing deployment scenarios
|
||||
- **Pipeable output** for CI/CD integration — the plan on stdout, everything else on stderr
|
||||
- **Session history** with replay capability
|
||||
- **Session history** for fast replay during development
|
||||
- **Multi-layered** config files with natural discovery and deterministic parameter resolution
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Load cubes** - Discovers and validates cubes from configured directories
|
||||
2. **Interactive prompts** - Select cubes, target host, and authentication method
|
||||
3. **Dependency resolution** - Topologically sorts cubes based on dependencies
|
||||
3. **Dependency resolution** - Resolves each cube's dependencies before the cube itself, so the deploy order is a topological order of the graph; a cycle is reported by name rather than recursed into
|
||||
4. **Variable assignment** - Validates and collects configuration with schema validation
|
||||
5. **Execute hooks** - Runs before/after hooks for orchestration
|
||||
6. **Deploy** - Sequentially executes pyinfra commands
|
||||
@@ -33,23 +35,20 @@ Nopy wraps pyinfra with structure, validation, and an interactive experience for
|
||||
|
||||
A cube is a **directory** containing two files:
|
||||
|
||||
- **JavaScript manifest**: `manifest.mjs` defining schema, dependencies, defaults, secrets, and hooks
|
||||
- **JavaScript manifest**: `manifest.mjs` defining schema, dependencies, defaults, secrets (encrypted only), and hooks
|
||||
- **Python deployment script**: `deploy.py`, a plain pyinfra script
|
||||
|
||||
Configuration variables are declared in the manifest and validated with Zod schemas before the deployment script runs.
|
||||
|
||||
```
|
||||
cubes/
|
||||
├── .npcubes
|
||||
└── apt/
|
||||
└── install/
|
||||
├── manifest.mjs
|
||||
└── deploy.py
|
||||
```
|
||||
|
||||
Any directory holding both files is treated as a cube, so cubes can be nested as deeply as you like to group them by topic. Discovery is recursive; directories starting with `.` and `node_modules` are skipped. Additional files in the cube directory (a `README.md`, config templates, and so on) are ignored by the loader and can be referenced from the deploy script — the script runs with its cube directory as the working directory.
|
||||
|
||||
The prefixed forms `<cube-name>.manifest.mjs` and `<cube-name>.deploy.py` are also still recognized, but plain `manifest.mjs` / `deploy.py` is the current convention.
|
||||
Any directory holding both files is treated as a cube, so cubes can be nested and grouped by topic. Discovery is recursive; hidden directories starting with `.` and `node_modules` are skipped. Additional files in the cube directory (a `README.md`, config templates, and so on) are ignored by the loader but can be referenced from the deploy script — ** the pyinfra script runs with its cube directory as the working directory**.
|
||||
|
||||
A cube's identity comes from the manifest's `id` field (see below). If `id` is omitted, nopy falls back to an `[id]` prefix in the manifest `name`, and finally to the directory's own name. Note that the id does not have to mirror the folder path — `cubes/network/tailscale` declares `id: 'net:tailscale'`.
|
||||
|
||||
@@ -65,7 +64,7 @@ export default cubes.Manifest({
|
||||
dependencies: () => [],
|
||||
schema: z.object({
|
||||
UPDATE: z.boolean().describe('Update package cache').default(false),
|
||||
PACKAGES: z.string().describe('Space-separated list of packages').default('vim htop'),
|
||||
PACKAGES: z.string().describe('Space-separated list of packages').default('curl htop'),
|
||||
})
|
||||
})
|
||||
```
|
||||
@@ -118,7 +117,7 @@ A variable can be set from several places in one run. Every assignment is kept,
|
||||
|
||||
This allows cubes to ship with reasonable defaults while still allowing users to override them globally via `.nopyrc.json` or interactively during deployment. Because `env` outranks the schema, `.nopyrc.json` is also what steers a run started with `--use-defaults`, which never prompts.
|
||||
|
||||
`prompt` and `param` rarely compete: a key a dependency supplies is left out of the prompt entirely, so the user is only ever asked about the keys nothing else has set.
|
||||
`prompt` and `param` rarely compete: a key supplied by a dependency is left out of the user input prompt entirely.
|
||||
|
||||
Ranking by origin rather than by arrival order is what makes replay work: a recorded value is applied *before* the cube would be prompted for, and prompting can still override it, but a `--data` value pushed in by a dependency is never clobbered by a stale recording.
|
||||
|
||||
@@ -572,7 +571,7 @@ A `--load-session` run *is* recorded, and the distinction is the point: a sessio
|
||||
|
||||
A run is *not* recorded when:
|
||||
|
||||
- `--dry-run`, `--print-only` or `--no-history` is passed — the first two deploy nothing, and history is what `-R` repeats
|
||||
- `--dry-run`, `--print-only` or `--no-save-history` is passed — the first two deploy nothing, and history is what `-R` repeats
|
||||
- No cubes were selected, so there was nothing to deploy
|
||||
- `history.autoSave` is set to `false` in `.nopyrc.json`
|
||||
- it is a `-R` or `-H` replay, as above
|
||||
|
||||
+48
-33
@@ -439,21 +439,30 @@ Recursive, per (cube, host):
|
||||
5. emit the deploy call;
|
||||
6. run `after` hooks.
|
||||
|
||||
There is no separate topological sort — the ordering falls out of the recursion,
|
||||
and a `${cubeId}:${host}` set makes emission idempotent. Consequently there is no
|
||||
cycle detection either: two mutually dependent cubes recurse until the stack
|
||||
overflows.
|
||||
There is no separate topological sort — emission is post-order, so a dependency
|
||||
is always emitted ahead of its dependent and the ordering *is* topological
|
||||
without an algorithm computing it. A `${cubeId}:${host}` set makes emission
|
||||
idempotent.
|
||||
|
||||
**Throws** when the cube id is unknown, when `useDefaults` cannot fill a required
|
||||
key, when a replay would need a value only the user has (secrets are never
|
||||
recorded), and when a cancelled prompt leaves a required key empty.
|
||||
Cycles are detected by the resolution stack rather than by the sort that does not
|
||||
exist: a (cube, host) pair re-entered while it is still resolving raises with the
|
||||
whole path named — `Circular dependency on host1: a → b → c → a`. The stack is
|
||||
separate from the idempotence set on purpose, since re-entering a *finished* cube
|
||||
with different `param` overrides is legitimate and a dependency or hook may do it.
|
||||
|
||||
The command it builds:
|
||||
**Throws** when the cube id is unknown, when the dependency graph contains a
|
||||
cycle, when `useDefaults` cannot fill a required key, when a replay would need a
|
||||
value only the user has (secrets are never recorded), and when a cancelled prompt
|
||||
leaves a required key empty.
|
||||
|
||||
The command it builds — an argv array, one element per argument, nothing quoted:
|
||||
|
||||
```
|
||||
pyinfra <host> -y [--user U --password P] --data "K=V" … --chdir <cubeDir> <cubeDir>/<deployScript>
|
||||
pyinfra <host> -y [-v|-vv|-vvv] [--debug] [--user U --password P] --data K=V … --chdir <cubeDir> <cubeDir>/<deployScript>
|
||||
```
|
||||
|
||||
The verbosity and debug flags come from `config.log` via `logConfigToFlags()`.
|
||||
|
||||
---
|
||||
|
||||
## Variables Module
|
||||
@@ -599,9 +608,14 @@ interface ExecutionOptions {
|
||||
### `executeDeployCalls(calls, options?)`
|
||||
|
||||
Runs the calls **sequentially**, in the order they were built, through
|
||||
`execa({ shell: true })` with `stdio: 'inherit'` so pyinfra's output reaches the
|
||||
terminal live. Stops at the first failure unless `continueOnError`. With
|
||||
`dryRun`, prints the plan and returns `[]` without executing.
|
||||
`execa(command[0], command.slice(1))` with `stdio: 'inherit'` so pyinfra's output
|
||||
reaches the terminal live. Stops at the first failure unless `continueOnError`.
|
||||
With `dryRun`, prints the plan and returns `[]` without executing.
|
||||
|
||||
**No shell.** It used to join `command` into one string and run it through
|
||||
`execa({ shell: true })`, which made every `--data` value shell syntax: a
|
||||
password or a variable containing `;`, a backtick or `$(…)` was executed rather
|
||||
than passed along. Spawning the argv directly removes the parse step entirely.
|
||||
|
||||
```typescript
|
||||
const results = await executeDeployCalls(calls, {
|
||||
@@ -630,9 +644,11 @@ maskVariables(call); // Record<string, string>
|
||||
|
||||
pyinfra takes its data on the command line, so the real values have to be in
|
||||
`call.command`; these are the last point before they would reach a log, a
|
||||
`--print-only` dump or a dry-run plan. `maskCommand` replaces the SSH
|
||||
`--password` argument and every `--data "KEY=…"` whose key the manifest declared
|
||||
a secret.
|
||||
`--print-only` dump or a dry-run plan. `maskCommand` walks the argv, replaces the
|
||||
element after `--password` and the value of every `--data KEY=…` whose key the
|
||||
manifest declared a secret, and shell-quotes the rest so `--print-only` output
|
||||
stays pasteable. It is the only thing that joins `command` into a string —
|
||||
nothing executes it that way.
|
||||
|
||||
This covers nopy's own output only. The value still reaches pyinfra on its
|
||||
command line, so it is visible in `ps` — inherent to pyinfra's `--data`
|
||||
@@ -808,7 +824,7 @@ interface SessionHistory {
|
||||
| `formatHistoryList(entries)` | `string` | what `nopy history` prints |
|
||||
|
||||
Recording is suppressed for a dry run, a print-only run, a `-R`/`-H` replay out
|
||||
of history, a run that built no deploy calls, `--no-history`, and
|
||||
of history, a run that built no deploy calls, `--no-save-history`, and
|
||||
`history.autoSave: false` in the config. A `--load-session` run **is** recorded: it is not in history already, and
|
||||
without the entry `-R` would have nothing to repeat.
|
||||
|
||||
@@ -870,9 +886,8 @@ config's `node_modules` rather than the working directory's. It is the same
|
||||
problem `PATH_PROPERTIES` solves for relative `cubeDirs`, with a different answer:
|
||||
a reference to resolve later instead of a rewritten path.
|
||||
|
||||
> The `CubePackageRef` name is currently not re-exported from the package root,
|
||||
> though `NopyConfig` refers to it. Import it from `@bitsquare/nopy` and you get
|
||||
> `NopyConfig` but not this type by name.
|
||||
Re-exported from the package root alongside `NopyConfig`, which refers to it —
|
||||
it was not, until the regeneration of this document noticed.
|
||||
|
||||
### `loadConfig()`
|
||||
|
||||
@@ -1105,7 +1120,7 @@ nopy install -l ./sess.json # replay a session file
|
||||
nopy install -n # dry run — print the plan, execute nothing
|
||||
nopy install -P # print the built pyinfra commands and exit
|
||||
nopy install -c # continue after a failure
|
||||
nopy install --no-history # do not record this run
|
||||
nopy install --no-save-history # do not record this run
|
||||
|
||||
nopy history # list recorded sessions (alias: h; -j for JSON)
|
||||
nopy clear-history # drop them all
|
||||
@@ -1125,8 +1140,10 @@ Exit code is 1 when any cube failed.
|
||||
"up to date", since an unanswerable check is not a negative answer. See
|
||||
[Known gaps](#known-gaps) for what that message conflates.
|
||||
|
||||
> `-H <id>` and `--no-history` share one Commander destination, so passing both
|
||||
> discards the id and falls through to an interactive run.
|
||||
> The suppression flag is `--no-save-history`, not `--no-history`. Commander
|
||||
> derives an option's destination from its long flag with `no-` stripped, so
|
||||
> `--no-history` wrote to the same `options.history` that `-H <id>` does and
|
||||
> `nopy install -H abc --no-history` silently discarded the id.
|
||||
|
||||
---
|
||||
|
||||
@@ -1169,12 +1186,15 @@ export default Manifest({
|
||||
});
|
||||
```
|
||||
|
||||
> **Call `.default()` before `.describe()`.** In zod 4, `.default()` returns a
|
||||
> `ZodDefault` wrapper that does not inherit `.description` from the type it
|
||||
> wraps, and the prompt reads the description off the outer node. So
|
||||
> `z.boolean().describe('Update cache').default(false)` prompts with the bare key
|
||||
> `UPDATE`, while `z.boolean().default(false).describe('Update cache')` prompts
|
||||
> with the sentence. Verified against zod 4.4.3.
|
||||
> **The order of `.default()` and `.describe()` does not matter.** It used to.
|
||||
> In zod 4, `.default()` returns a `ZodDefault` wrapper that does not inherit
|
||||
> `.description` from the type it wraps, so
|
||||
> `z.boolean().describe('Update cache').default(false)` prompted with the bare
|
||||
> key `UPDATE` while the other order prompted with the sentence — a difference
|
||||
> nothing announced, and one that 15 of the 22 core cubes were on the wrong side
|
||||
> of. The prompt now unwraps `default`/`optional`/`nullable` looking for a
|
||||
> description, so either chaining order gives the label. Verified against
|
||||
> zod 4.4.3.
|
||||
|
||||
Every schema key reaches pyinfra as `--data KEY=value`, so `host.data.KEY` is
|
||||
always defined. pyinfra parses the values itself: `"true"` arrives as a bool and
|
||||
@@ -1206,11 +1226,6 @@ For packaging cubes as an installable npm bundle, see
|
||||
Real behaviour that a reader would otherwise take on trust. Tracked in
|
||||
`DOCS-AUDIT.md` and summarised in `CLAUDE.md`.
|
||||
|
||||
- **`logConfigToFlags()` is never consumed.** It is exported and unit-tested, but
|
||||
nothing feeds its output into the built pyinfra command, so `log.verbosity` and
|
||||
`log.debug` in `.nopyrc.json` have no effect today.
|
||||
- **No cycle detection.** Ordering is a side effect of recursion, not a
|
||||
topological sort. Two mutually dependent cubes overflow the stack.
|
||||
- **`DeployCall.dependencies` is always `[]`.** The field is populated nowhere;
|
||||
dependency information lives in the emission order.
|
||||
- **`ExecutionResult.stdout` / `.stderr` are always `undefined`,** because the
|
||||
|
||||
@@ -10,6 +10,7 @@ export type { Assignment, Origin, TVariables, Value } from './nopy.common.js';
|
||||
// Variables
|
||||
export { MASK, Variable, Variables } from './nopy.common.js';
|
||||
export type {
|
||||
CubePackageRef,
|
||||
ExecutionConfig,
|
||||
HistoryConfig,
|
||||
LogConfig,
|
||||
|
||||
Reference in New Issue
Block a user