[refactor] moving cubes into own package"

[fix] default parameter run records parameters in session for replay[fix] remove default parameters for several cubes
2026-07-28 12:18:10 +02:00
parent ac050c4459
commit 6ecb2c366f
130 changed files with 3386 additions and 520 deletions
@@ -33,7 +33,7 @@ Nopy wraps pyinfra with structure, validation, and an interactive experience for

 A cube is a **directory** containing two files:

- **JavaScript manifest**: `manifest.mjs` defining schema, dependencies, defaults, and hooks
+- **JavaScript manifest**: `manifest.mjs` defining schema, dependencies, defaults, secrets, and hooks
 - **Python deployment script**: `deploy.py`, a plain pyinfra script

 Configuration variables are declared in the manifest and validated with Zod schemas before the deployment script runs.
@@ -96,7 +96,7 @@ apt.packages(
 )
 ```

-Every key defined in the manifest `schema` is guaranteed to be present on `host.data` — either from the Zod `.default()`, from `.nopyrc.json`, from a dependency, or from a user prompt.
+Every key defined in the manifest `schema` is guaranteed to be present on `host.data` — either from the Zod `.default()`, from `.nopyrc.json`, from a recorded session, from a dependency, or from a user prompt.

 **Value types**: pyinfra parses `--data` values before your script sees them. `"true"` / `"false"` become booleans, numeric strings become `int`, valid JSON becomes the parsed structure, and everything else stays a string. This is why `UPDATE` can be handed straight to pyinfra's `update=` argument, while `PACKAGES` is wrapped in `str(...)` before splitting.

@@ -104,18 +104,57 @@ Every key defined in the manifest `schema` is guaranteed to be present on `host.

 Variable defaults are defined directly in the Zod schema using `.default()`. This ensures that every cube has a predictable starting state and provides type-safe default values.

-**Priority order (lowest to highest):**
+A variable can be set from several places in one run. Every assignment is kept, tagged with where it came from — its **origin** — and the highest-ranked origin wins.

-1. Zod schema `.default()` values
-2. Global `env` from `.nopyrc.json`
-3. User prompts, or the recorded answers on session replay
-4. Variables passed in by a dependency or a hook
+**Origins, lowest to highest:**
+
+| Origin | Set by |
+| --------- | ------------------------------------------------------- |
+| `default` | the Zod schema's `.default()` |
+| `env` | the `env` block of `.nopyrc.json` |
+| `session` | a value recorded in a session file or history entry |
+| `prompt` | what the user typed |
+| `param` | a dependency spec or a `before`/`after` hook |

 This allows cubes to ship with reasonable defaults while still allowing users to override them globally via `.nopyrc.json` or interactively during deployment. Because `env` outranks the schema, `.nopyrc.json` is also what steers a run started with `--use-defaults`, which never prompts.

-3 and 4 rarely compete: a key a dependency supplies is left out of the prompt entirely, so the user is only ever asked about the keys nothing else has set.
+`prompt` and `param` rarely compete: a key a dependency supplies is left out of the prompt entirely, so the user is only ever asked about the keys nothing else has set.

-A field declared without `.default()` has none of sources 1 and 2 to fall back on. It is prompted for like any other, with an empty initial value — but a run that cannot prompt (`--use-defaults`) fails on it unless `env` or a dependency provides it.
+Ranking by origin rather than by arrival order is what makes replay work: a recorded value is applied *before* the cube would be prompted for, and prompting can still override it, but a `--data` value pushed in by a dependency is never clobbered by a stale recording.
+
+A field declared without `.default()` has no `default` origin to fall back on. It is prompted for like any other, with an empty initial value — but a run that cannot prompt (`--use-defaults`) fails on it unless `env` or a dependency provides it.
+
+#### Secrets
+
+A manifest can name schema keys that hold sensitive values:
+
+```javascript
+export default cubes.Manifest({
+    id: 'user:add',
+    name: 'Add a user account',
+    secrets: ['PASSWORD'],
+    schema: z.object({
+        USERNAME: z.string().describe('Username for the new account').default('deploy'),
+        PASSWORD: z.string().describe('Password for the new user account').default('changeme'),
+    })
+})
+```
+
+Every entry must be a key of `schema`; naming anything else is a manifest error and aborts the run, so a typo fails loudly instead of silently leaving a value unprotected.
+
+Declaring a key a secret changes three things:
+
+- **It is never written to a session file or to the history.** Everything else the run settled on is recorded — including values that came from a `.default()` — but declared secrets are left out.
+- **It is masked wherever a command or a plan is printed** — `--dry-run`, `--print-only`, and the debug log all show `********` in place of the value, in the variable list *and* in the `pyinfra` command line above it. The SSH password passed via `--password` is masked the same way, whether or not any cube declares secrets.
+- **It is re-prompted on replay**, since there is nothing recorded to replay from (see [Session Recording and Replay](#session-recording-and-replay)).
+
+Nopy does not guess. A key called `PASSWORD` in a manifest that declares no `secrets` is treated as an ordinary variable — recorded, and printed in the clear.
+
+Three limits are worth knowing, because `secrets` keeps a value out of the files nopy writes and nothing more:
+
+- **It is on the command line.** pyinfra takes its data as `--data KEY=value`, so the real value is visible in `ps` for as long as the deployment runs. Masking covers nopy's own output, not the process table.
+- **The prompt shows it.** The variable form displays and pre-fills what it is asking about, so a secret is on screen while it is being entered or confirmed.
+- **A `.default()` is not protected.** A default lives in the manifest, in plain text, wherever the manifest is checked in. Give a secret a placeholder default like `changeme` if it needs one at all, never a real credential.

 ### Configuration

@@ -125,6 +164,7 @@ Uses `.nopyrc.json` files (project-level or home directory) containing:
 {
  "hosts": ["host1.example.com", "host2.example.com"],
  "cubeDirs": ["./cubes", "../shared-cubes"],
+  "cubePackages": ["@bitsquare/cubes-core"],
  "env": {
    "SHARED_VAR": "value"
  },
@@ -144,6 +184,8 @@ Uses `.nopyrc.json` files (project-level or home directory) containing:

 `history` controls automatic session recording (see [Deployment History](#deployment-history)), and `execution.continueOnError` sets the default for `--continue-on-error`.

+`cubeDirs` holds paths, `cubePackages` holds installed npm packages that ship cubes — see [Cube Discovery](#cube-discovery) below and [CUBE-BUNDLES.md](docs/CUBE-BUNDLES.md) for publishing your own. Both are additive, and both resolve relative to the config file that named them, not to the working directory: a `.nopyrc.json` two levels up may name a package that only exists in *its* `node_modules`.
+
 #### Logging Configuration

 Control pyinfra output verbosity and debug information using the `log` configuration object:
@@ -213,12 +255,16 @@ Sessions are stored in `.nopysession.json` files with the following structure:

 **Structure Details:**

- **`cubes`**: Array of cubes with only cube-specific variables (not global env vars)
- **`env`**: Global environment variables shared across cubes (like in `.nopyrc.json`)
+- **`cubes`**: Array of cubes with the variable values that cube ran with
+- **`env`**: The `env` block of `.nopyrc.json` as it stood at record time, kept for reference
 - **`hosts`**: Array of target hosts
 - **`auth`**: Authentication configuration (passwords are never stored)

-**Security Note**: Passwords are never stored in session files. If a session uses password authentication, you'll be prompted for the password during replay.
+**What is recorded:** every value each cube settled on, regardless of where it came from — a value the user typed, one inherited from `.nopyrc.json` `env`, one a dependency supplied, and one that fell through to the schema's `.default()` are all written out the same way. A session is therefore a full snapshot rather than a diff, and a `--use-defaults` run produces a session with real values in it instead of an empty one.
+
+The consequence is that replay is faithful rather than re-derived: the recorded value outranks the current `.nopyrc.json` `env` and the current schema default, so editing either one does not silently change what a replay does. To pick up a new default, record a fresh session.
+
+**Security Note**: Passwords are never stored in session files. This covers both the SSH password — a session records the auth *method* and username, never the credential — and any schema key a cube's manifest lists under [`secrets`](#secrets). Both are re-prompted on replay.

 #### Recording a Session

@@ -240,12 +286,48 @@ nopy install --load-session my-deployment.nopysession.json
 # Only password authentication will prompt for credentials
 ```

+A replay runs straight through without asking anything, with three exceptions. Password authentication always re-prompts. A session with no recorded host falls back to the host picker. And a cube is re-prompted for its declared secrets, plus for any required variable the session has no value for — which happens when the cube's schema has gained a field since the session was written.
+
+Those re-prompts are what a session cannot supply, so `--use-defaults` cannot paper over them: combining `-D` with a replay that needs either fails with a message naming the keys rather than deploying with a placeholder. Put the values under `env` in `.nopyrc.json` to make such a replay unattended.
+
 ### Cube Discovery

 Nopy searches for cubes in:

 1. Directories specified in `.nopyrc.json` `cubeDirs`
-2. Directories containing a `.npcubes` marker file (searching upwards from current directory)
+2. Cube directories of every package listed in `.nopyrc.json` `cubePackages`
+3. Directories containing a `.npcubes` marker file (searching upwards from current directory)
+
+All three are unioned and scanned the same way. A directory is a cube when it holds both a manifest (`manifest.mjs` or `*.manifest.mjs`) and a deploy script (`deploy.py` or `*.deploy.py`); dotted directories and `node_modules` are skipped during the scan.
+
+#### Cube packages
+
+A cube package is an ordinary npm package that ships cube directories and points at them from its own `package.json`:
+
+```json
+{
+  "name": "@bitsquare/cubes-core",
+  "nopy": { "cubes": ["./cubes"] }
+}
+```
+
+Install it and name it — nothing needs linking or copying:
+
+```sh
+pnpm add -D @bitsquare/cubes-core
+```
+
+```json
+{ "cubePackages": ["@bitsquare/cubes-core"] }
+```
+
+Naming a package is a statement that cubes are expected from it, so anything wrong is an error that aborts the run rather than a silent skip: the package is not installed, it declares no `nopy.cubes`, or an entry points at a directory that does not exist or lies outside the package.
+
+#### Ids are claimed globally
+
+A cube id such as `apt:essentials` is claimed across every source at once, not per directory or per package. Two cubes with the same id abort the run with an error naming both and where each came from. There is no precedence rule and no shadowing — a local cube does not quietly win over a packaged one, in either direction. Prefix your own cubes distinctly if you point `cubeDirs` at a local tree alongside an installed bundle.
+
+Writing cubes to publish is covered in [CUBE-BUNDLES.md](docs/CUBE-BUNDLES.md).

 ## Command Line Usage

@@ -319,6 +401,17 @@ have no default values. Set them under "env" in .nopyrc.json, pass them from a
 dependency, or drop --use-defaults to be prompted.
 ```

+Pairing `-D` with a replay fails the same way when the replay would have to ask
+something — a declared [secret](#secrets), which is never recorded, or a required
+variable the session has no value for. Both are the sources `-D` has no substitute
+for, so it stops rather than deploying a placeholder:
+
+```
+Error: Cube "user:add" cannot be replayed with --use-defaults: PASSWORD would
+have to be entered. Secrets are never recorded in a session. Replay without
+--use-defaults, or set the values under "env" in .nopyrc.json.
+```
+
 **Use SSH key authentication**:

 ```bash
@@ -434,20 +527,20 @@ Session History:
 Total: 2 session(s)
 ```

-Each entry records the selected cubes together with the variable values that were answered at the prompts, the target hosts, the authentication method, and the username — never the password. Pass an ID to `-H` to run that exact combination again:
+Each entry records the selected cubes together with every variable value they ran with, the target hosts, the authentication method, and the username — never the password, and never a key the manifest declared a [secret](#secrets). Pass an ID to `-H` to run that exact combination again:

 ```bash
 nopy install -H mdk0zzp8b71cq
 ```

-A replay is non-interactive: cube selection, host, and variable values all come from the entry, so nopy runs straight through without asking anything. The two exceptions are password authentication, which always re-prompts, and an entry with no recorded host, which falls back to the host picker.
+A replay is non-interactive: cube selection, host, and variable values all come from the entry, so nopy runs straight through without asking anything. It asks only for what the entry cannot hold — the password under password authentication, and any declared secret — plus the host picker when the entry recorded none.

 Two things are worth knowing before relying on an older entry:

- **Recorded values are applied as defaults, not as a frozen snapshot.** If a cube's schema has gained a variable since the run was recorded, the replay neither prompts for it nor fails — the new variable quietly takes its Zod `.default()`. Global `env` values are likewise read from the *current* `.nopyrc.json` rather than from the entry.
+- **Recorded values win over the current configuration.** The entry is a snapshot of everything the run settled on, so editing a cube's `.default()` or the `env` block of `.nopyrc.json` afterwards does not change what the replay does. A variable the schema has gained *since* the entry was written has nothing recorded: if it has a `.default()` the replay quietly takes it, and if it is required the replay prompts for it.
 - **A replay fails if a cube no longer exists.** Renaming or deleting a cube id makes every history entry that referenced it unreplayable: nopy logs `Cube from session not found` and then aborts with `Cube not found: <id>`.

-The history lives in `.nopy.history.json` in the working directory and uses the same structure as a session file, so trimming the array by hand is a perfectly good way to prune it. It does contain the variable values that were entered, which is why it is listed in this repository's `.gitignore` — treat it like any other file holding deployment configuration. A corrupt or unreadable history file is treated as empty rather than raising an error, which looks exactly like a project that has never been deployed from.
+The history lives in `.nopy.history.json` in the working directory and uses the same structure as a session file, so trimming the array by hand is a perfectly good way to prune it. It does contain the variable values a run used, which is why it is listed in this repository's `.gitignore` — treat it like any other file holding deployment configuration. A corrupt or unreadable history file is treated as empty rather than raising an error, which looks exactly like a project that has never been deployed from.

 For a run you want to keep indefinitely, don't rely on history — it rotates. Use `--save-session` to write it to a file you control (see [Session Recording and Replay](#session-recording-and-replay)).

@@ -468,7 +561,9 @@ npm run debug
 ## Documentation

 - [Cube Hooks](docs/HOOKS.md) - Lifecycle hooks for dynamic orchestration
+- [Cube Bundles](docs/CUBE-BUNDLES.md) - Distributing cubes as npm packages
 - [Session Format](docs/SESSION_FORMAT.md) - Internal JSON/MJS session structure
+- [API Reference](docs/API.md) - Types and exported functions

 ## Resources