hardening and bugfixing prior to stable release
This commit is contained in:
Benjamin Diedrichsen
+70
-30
@@ -137,6 +137,7 @@ class Cube<Schema extends AnyObjectSchema = AnyObjectSchema> {
|
||||
get secrets(): string[]; // manifest.secrets ?? []
|
||||
|
||||
getDefaults(): z.infer<Schema>;
|
||||
schemaKeys(): string[];
|
||||
requiredKeys(): string[];
|
||||
isSecret(key: string): boolean;
|
||||
}
|
||||
@@ -151,6 +152,11 @@ single required field used to leave the cube with no variables at all.
|
||||
and not optional. A `--use-defaults` run that cannot supply one aborts by name
|
||||
rather than deploying the cube with the value missing.
|
||||
|
||||
`schemaKeys()` returns every declared key, required or not. It answers a
|
||||
different question — whether the cube *claims to know about* a key, rather than
|
||||
whether it has a value for one — and that is what decides whether a secret in the
|
||||
config `env` is allowed to reach it.
|
||||
|
||||
### `CubeSource`
|
||||
|
||||
Where a cube came from. Carried because a cube's directory does not say how it
|
||||
@@ -264,13 +270,12 @@ const result = await nopy({ useDefaults: true, dryRun: true });
|
||||
|------|------|---------|-------------|
|
||||
| `useDefaults` | `boolean` | `false` | Skip the variable prompts. A cube with a required key nothing supplied aborts the run by name. |
|
||||
| `useAuthKey` | `boolean` | `false` | Force SSH key auth, skipping the auth prompt. |
|
||||
| `saveSession` | `string` | – | Path to write the session to. **Ignored during a replay.** |
|
||||
| `saveSession` | `string` | – | Path to write the session to. Honoured on a replay too. |
|
||||
| `loadSession` | `string` | – | Path to a session file to replay. |
|
||||
| `replaySession` | `NopySession` | – | A session object to replay, used by `-R` / `-H` from history. Takes precedence over `loadSession`. |
|
||||
| `dryRun` | `boolean` | `false` | Print the execution plan instead of running it. |
|
||||
| `printOnly` | `boolean` | `false` | Print the built pyinfra commands and return; the executor is never reached. |
|
||||
| `continueOnError` | `boolean` | `false` | Keep going after a cube fails. |
|
||||
| `jsonOutput` | `boolean` | `false` | Suppress the config banner and progress lines. See [Known gaps](#known-gaps). |
|
||||
| `saveToHistory` | `boolean` | `true` | Record the session in `.nopy.history.json`. |
|
||||
|
||||
**Returns:** `Promise<NopyResult | undefined>` — `undefined` when cube loading
|
||||
@@ -507,9 +512,10 @@ displaced stays visible underneath. The trace is never persisted.
|
||||
|
||||
```typescript
|
||||
class Variables {
|
||||
constructor(env?: TVariables);
|
||||
constructor(env?: TVariables, globalSecrets?: Iterable<string>);
|
||||
|
||||
declareSecrets(cube: string, keys: readonly string[]): void;
|
||||
declareSchema(cube: string, keys: readonly string[]): void;
|
||||
isSecret(cube: string, name: string): boolean;
|
||||
|
||||
assign(cube: string, origin: Origin, values?: TVariables): void;
|
||||
@@ -527,6 +533,25 @@ const MASK = '********';
|
||||
`declareSecrets()` is retroactive as well as prospective, so it does not matter
|
||||
whether the caller declares before or after the values arrive.
|
||||
|
||||
`globalSecrets` is every key *any* manifest declares secret, plus the config's
|
||||
own `secrets` list. `nopy()` computes it once after `loadCubes()`, before the
|
||||
first cube resolves, so resolution order cannot change whether a value is treated
|
||||
as a credential. It does two things:
|
||||
|
||||
- `isSecret()` is true for such a key on **every** cube, so a manifest that lists
|
||||
`PASSWORD` in `schema` and forgets it in `secrets` still gets masking and still
|
||||
keeps the value out of the session.
|
||||
- The config `env` stops being broadcast for it. Ordinary `env` keys are seeded
|
||||
onto every cube — deliberately, since a cube may read a key off `host.data`
|
||||
that it never declared — but a secret reaches only the cubes whose
|
||||
`schemaKeys()` include it.
|
||||
|
||||
`declareSchema()` is what supplies those keys, and it has an ordering
|
||||
requirement: call it before anything assigns to the cube, because the first
|
||||
assignment is what seeds `env`. `BuildContext.resolveCube` calls it immediately
|
||||
after `declareSecrets()`. It deliberately does not create the cube's bucket
|
||||
itself.
|
||||
|
||||
`persistable()` leaves a secret out entirely rather than masking it, so a replay
|
||||
sees it as absent and asks for it again. That is why replaying a session whose
|
||||
cubes declare secrets is interactive even under `-D` — a `-D` replay that would
|
||||
@@ -585,15 +610,16 @@ const results = await executeDeployCalls(calls, {
|
||||
});
|
||||
```
|
||||
|
||||
### `outputExecutionPlan(calls, asJson?)`
|
||||
### `outputExecutionPlan(calls)`
|
||||
|
||||
```typescript
|
||||
outputExecutionPlan(deployCalls); // text
|
||||
outputExecutionPlan(deployCalls, true); // JSON
|
||||
outputExecutionPlan(deployCalls);
|
||||
```
|
||||
|
||||
Both forms mask secrets. Note that `executeDeployCalls` calls this without the
|
||||
second argument, so `--dry-run --json` prints the text plan.
|
||||
Prints the plan a `--dry-run` shows, with secrets masked. Went from
|
||||
`(calls, asJson?)` to `(calls)` when `--json` was removed; the JSON branch was
|
||||
unreachable from the CLI, since `executeDeployCalls` never passed the second
|
||||
argument.
|
||||
|
||||
### `maskCommand(call)` / `maskVariables(call)`
|
||||
|
||||
@@ -641,7 +667,7 @@ interface WorkflowResult {
|
||||
authMethod: string;
|
||||
username?: string;
|
||||
password?: string;
|
||||
isReplay: boolean;
|
||||
replaySource?: 'file' | 'history'; // undefined on a fresh interactive run
|
||||
}
|
||||
|
||||
interface WorkflowOptions {
|
||||
@@ -673,10 +699,12 @@ The same, from a session object rather than a path — the `-R` / `-H` path.
|
||||
|
||||
```typescript
|
||||
interface NopySession {
|
||||
cubes: CubeSession[]; // required
|
||||
auth: AuthSession; // required
|
||||
version?: string;
|
||||
timestamp?: string; // ISO 8601
|
||||
name?: string;
|
||||
cubes: CubeSession[];
|
||||
hosts?: string[];
|
||||
auth: AuthSession;
|
||||
env?: TVariables;
|
||||
}
|
||||
|
||||
@@ -692,7 +720,10 @@ interface AuthSession {
|
||||
}
|
||||
```
|
||||
|
||||
There is no `version` or `timestamp` field, and nothing validates compatibility.
|
||||
`version` and `timestamp` are stamped on every session nopy writes and demanded
|
||||
of none it reads — an older file, or a hand-written one, simply lacks them.
|
||||
Nothing validates compatibility beyond a warning on an unrecognised `version`;
|
||||
the constant is exported as `SESSION_VERSION`.
|
||||
|
||||
A `CubeSession` records every value the cube settled on, whatever its origin —
|
||||
not just the prompted ones — minus anything the manifest declared a secret. So a
|
||||
@@ -702,8 +733,7 @@ and `env` happen to say later.
|
||||
|
||||
### `saveSession(session, filePath)`
|
||||
|
||||
Writes JSON, creating the directory if needed. Note that `nopy()` skips this
|
||||
during a replay.
|
||||
Writes JSON, creating the directory if needed.
|
||||
|
||||
### `loadSession(filePath)`
|
||||
|
||||
@@ -713,7 +743,8 @@ const session = await loadSession('./deployment.session.mjs'); // default expor
|
||||
```
|
||||
|
||||
Dispatches on the extension; `.json` and `.mjs` only. Validates that `cubes` is
|
||||
an array, that `hosts` (if present) is an array, and that `auth` exists.
|
||||
an array, that `hosts` (if present) is an array, and that `auth` exists. A
|
||||
`version` other than `SESSION_VERSION` warns on stderr and loads anyway.
|
||||
|
||||
### `createSession(params)`
|
||||
|
||||
@@ -725,18 +756,27 @@ const session = createSession({
|
||||
});
|
||||
```
|
||||
|
||||
Stamps `version` and `timestamp`; pass `timestamp` to override the latter. It
|
||||
does not derive a `name` — that needs the resolved cube list, which does not
|
||||
exist yet at the point the session is created, so `nopy()` fills it in at save
|
||||
time.
|
||||
|
||||
### `describeSession(session, timestamp)`
|
||||
|
||||
The one-line `date - cubes → hosts` description, shared with the history list so
|
||||
that the two cannot drift.
|
||||
|
||||
### `listSessions(dirPath?)`
|
||||
|
||||
Non-recursive; matches **`*.session.json`** and **`*.session.mjs`** only.
|
||||
A file named `deploy.nopysession.json` will not be listed, though `loadSession`
|
||||
reads it fine.
|
||||
Non-recursive; matches `*.nopysession.json`, `*.nopysession.mjs`,
|
||||
`*.session.json` and `*.session.mjs`.
|
||||
|
||||
---
|
||||
|
||||
## History Module
|
||||
|
||||
Sessions are recorded automatically after a successful non-replay run, into
|
||||
`.nopy.history.json` in the working directory.
|
||||
Sessions are recorded automatically, into `.nopy.history.json` in the working
|
||||
directory, before the deploy commands run — so a failed run is recorded too.
|
||||
|
||||
```typescript
|
||||
const HISTORY_FILE = '.nopy.history.json';
|
||||
@@ -767,8 +807,10 @@ interface SessionHistory {
|
||||
| `removeFromHistory(id)` | `boolean` | `false` if the id was not found |
|
||||
| `formatHistoryList(entries)` | `string` | what `nopy history` prints |
|
||||
|
||||
Recording is suppressed for a dry run, a replay, a run that built no deploy
|
||||
calls, `--no-history`, and `history.autoSave: false` in the config.
|
||||
Recording is suppressed for a dry run, a print-only run, a `-R`/`-H` replay out
|
||||
of history, a run that built no deploy calls, `--no-history`, and
|
||||
`history.autoSave: false` in the config. A `--load-session` run **is** recorded: it is not in history already, and
|
||||
without the entry `-R` would have nothing to repeat.
|
||||
|
||||
---
|
||||
|
||||
@@ -785,6 +827,7 @@ interface NopyConfig {
|
||||
cubeDirs: string[];
|
||||
cubePackages: CubePackageRef[];
|
||||
env: TVariables;
|
||||
secrets?: string[]; // env keys to treat as sensitive that no manifest declares
|
||||
log?: LogConfig;
|
||||
history?: HistoryConfig;
|
||||
execution?: ExecutionConfig;
|
||||
@@ -1031,7 +1074,7 @@ on `PATH`.
|
||||
**never throws** — it sits in front of every command the user actually asked
|
||||
for. Returns `null` immediately when `isUpdateCheckDisabled(env)`:
|
||||
`NOPY_NO_UPDATE_CHECK` set to anything but `0`/`false`, or `CI` set at all. The
|
||||
CLI prints it to **stderr**, so `--json` and piped stdout stay clean.
|
||||
CLI prints it to **stderr**, so a piped `--print-only` stays clean.
|
||||
|
||||
### `selfUpdate(options)` → `SelfUpdateResult`
|
||||
|
||||
@@ -1062,7 +1105,6 @@ nopy install -l ./sess.json # replay a session file
|
||||
nopy install -n # dry run — print the plan, execute nothing
|
||||
nopy install -P # print the built pyinfra commands and exit
|
||||
nopy install -c # continue after a failure
|
||||
nopy install -j # JSON output
|
||||
nopy install --no-history # do not record this run
|
||||
|
||||
nopy history # list recorded sessions (alias: h; -j for JSON)
|
||||
@@ -1167,17 +1209,15 @@ Real behaviour that a reader would otherwise take on trust. Tracked in
|
||||
- **`logConfigToFlags()` is never consumed.** It is exported and unit-tested, but
|
||||
nothing feeds its output into the built pyinfra command, so `log.verbosity` and
|
||||
`log.debug` in `.nopyrc.json` have no effect today.
|
||||
- **`--json` emits nothing on success.** `jsonOutput` suppresses the banner and
|
||||
the progress lines, and prints `{success: false, errors}` when cube *loading*
|
||||
fails. The success path returns `NopyResult` to the caller without printing it,
|
||||
so a CI job gets pyinfra's inherited stdio and an exit code. `--dry-run --json`
|
||||
prints the *text* plan.
|
||||
- **No cycle detection.** Ordering is a side effect of recursion, not a
|
||||
topological sort. Two mutually dependent cubes overflow the stack.
|
||||
- **`DeployCall.dependencies` is always `[]`.** The field is populated nowhere;
|
||||
dependency information lives in the emission order.
|
||||
- **`ExecutionResult.stdout` / `.stderr` are always `undefined`,** because the
|
||||
executor inherits stdio rather than capturing it.
|
||||
executor inherits stdio rather than capturing it. This is also why `install`
|
||||
has no `--json`: during a run nopy does not own its own stdout, so there is no
|
||||
stream to put a machine-readable answer on. Use `--print-only` for the plan and
|
||||
the exit code for the verdict.
|
||||
- **Hook variables are not schema-validated.** The second argument to a hook is
|
||||
the effective values as collected. `schema.parse()` runs in exactly one place —
|
||||
`Cube.getDefaults()`, against `{}` — and prompt input is type-coerced, which is
|
||||
|
||||
@@ -314,8 +314,8 @@ Rename one of them, or remove a source from .nopyrc.json.
|
||||
There is deliberately no override, alias or precedence rule. If two bundles ever
|
||||
claim the same id they are mutually exclusive, and the fix is upstream.
|
||||
|
||||
Surface the source in the interactive picker and in `--json` output so a user can
|
||||
see where a cube came from before running it.
|
||||
Surface the source in the interactive picker so a user can see where a cube came
|
||||
from before running it.
|
||||
|
||||
## Phase 4 — `@bitsquare/nopy-cubes`, the authoring package — **done**
|
||||
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
# Requirements and facts
|
||||
|
||||
A proposal, not a record. Nothing here is built.
|
||||
|
||||
The question it answers: a cube should be able to say *"in order to run, a user
|
||||
named xyz must exist, with fish as its shell, in these groups"* — and the engine
|
||||
should check that against the real host rather than blindly running whatever
|
||||
cube happens to create such a user.
|
||||
|
||||
Everything under [What pyinfra actually gives us](#what-pyinfra-actually-gives-us)
|
||||
was measured against pyinfra **3.5.1**. Everything under
|
||||
[The proposal](#the-proposal) is design.
|
||||
|
||||
## Contents
|
||||
|
||||
- [The problem with `dependencies`](#the-problem-with-dependencies)
|
||||
- [What pyinfra actually gives us](#what-pyinfra-actually-gives-us)
|
||||
- [The proposal](#the-proposal)
|
||||
- [Why `facts.py` exposes a function, not a script](#why-factspy-exposes-a-function-not-a-script)
|
||||
- [Execution model](#execution-model)
|
||||
- [Sharp edges](#sharp-edges)
|
||||
- [Measured vs. assumed](#measured-vs-assumed)
|
||||
|
||||
---
|
||||
|
||||
## The problem with `dependencies`
|
||||
|
||||
`manifest.dependencies` conflates two things that are not the same:
|
||||
|
||||
- **a requirement** — "this cube needs a user xyz to exist"
|
||||
- **a remedy** — "therefore run `user:add`"
|
||||
|
||||
Today only the remedy is expressible. `user:add` declares
|
||||
`dependencies: () => ['apt:essentials']`, which means *always run
|
||||
`apt:essentials`*, on every host, forever, regardless of whether anything it
|
||||
installs is missing.
|
||||
|
||||
### What that actually costs
|
||||
|
||||
Be honest about this, because it is smaller than it first looks and it changes
|
||||
what the feature is for.
|
||||
|
||||
pyinfra operations are **already fact-diffed**. `server.user(present=True,
|
||||
shell=…)` gathers `server.Users` itself and no-ops when the state matches. The
|
||||
current design is therefore not *incorrect* — it is idempotent. What it costs is:
|
||||
|
||||
1. **Prompts.** Resolving `user:add` drags `apt:essentials` into the plan *and
|
||||
its variables into the prompt sequence*. The user is asked questions about a
|
||||
cube they never chose. This is the real, visible tax.
|
||||
2. **Time.** A no-op pyinfra run is still a connection, a fact gather, and an
|
||||
operation build.
|
||||
3. **Legibility.** The plan cannot say *"user xyz already exists, skipping"*.
|
||||
|
||||
This matters because the obvious objection to the whole feature is *"just write
|
||||
`if not host.get_fact(…)` inside `deploy.py` — that is idiomatic pyinfra"*. The
|
||||
answer is that nopy's layer is **planning and prompting**: deciding which cubes
|
||||
enter the plan and which variables to ask for, before anything runs. A condition
|
||||
inside `deploy.py` cannot help with either. That is the justification for lifting
|
||||
facts into the Node runtime at all — and it means the feature buys **legibility
|
||||
and prompt-avoidance, not correctness**.
|
||||
|
||||
## What pyinfra actually gives us
|
||||
|
||||
### The `fact` subcommand is not a machine interface
|
||||
|
||||
`pyinfra @local fact server.Groups` prints JSON — and prints it to **stderr**,
|
||||
via `click.echo(jsonify(…), err=True)` in `pyinfra_cli/prints.py:print_fact`,
|
||||
interleaved with `--> Loading config...` progress lines. Worse,
|
||||
`_run_fact_operations` wraps each fact in `except PyinfraError: pass`, so the
|
||||
command **exits 0 whether or not the fact resolved**.
|
||||
|
||||
Parsing that stream means fishing a JSON blob out of styled log output and
|
||||
having no exit code to check. Rejected.
|
||||
|
||||
### A deploy file that prints to stdout is clean
|
||||
|
||||
pyinfra deploy files execute on the control machine at *build* time, and
|
||||
`host.get_fact()` is available there — it is exactly how operations do their own
|
||||
diffing. A file that gathers facts, prints JSON, and declares no operations
|
||||
works:
|
||||
|
||||
```python
|
||||
import json, sys
|
||||
from pyinfra import host
|
||||
from pyinfra.facts.server import Users, Which
|
||||
|
||||
users = host.get_fact(Users)
|
||||
u = users.get(host.data.USER)
|
||||
print("###NOPY-FACTS###" + json.dumps({
|
||||
"host": host.name,
|
||||
"exists": u is not None,
|
||||
"shell": (u or {}).get("shell"),
|
||||
"groups": (u or {}).get("groups", []),
|
||||
}), file=sys.stdout)
|
||||
```
|
||||
|
||||
```
|
||||
$ pyinfra @local -y --data USER=someone facts.py
|
||||
EXIT=0
|
||||
--- STDOUT ---
|
||||
###NOPY-FACTS###{"host": "@local", "exists": false, "shell": null, "groups": []}
|
||||
--- STDERR ---
|
||||
--> Loading config...
|
||||
…
|
||||
--> Results:
|
||||
Operation Hosts Success Error No Change
|
||||
Grand total - - - -
|
||||
```
|
||||
|
||||
Confirmed properties:
|
||||
|
||||
- **stdout is exclusively ours.** Every byte pyinfra emits goes to stderr, which
|
||||
is the same reason nopy's own logging goes there (`configureLogtape`).
|
||||
- **`--data` flows in identically** to a deploy script. Parameterising a probe
|
||||
is free.
|
||||
- **Zero operations is legal.** `Grand total -`, exit 0.
|
||||
- **`host.name` is available**, so multi-host output is self-tagging.
|
||||
- **A custom `FactBase` subclass in a sibling module imports fine** under
|
||||
`--chdir`, so a cube can ship fact classes pyinfra does not have.
|
||||
- **An exception exits 1** with a traceback on stderr.
|
||||
|
||||
### The trap
|
||||
|
||||
**An unreachable host exits 0 and prints nothing.**
|
||||
|
||||
```
|
||||
$ pyinfra nonexistent.invalid.example -y probe.py
|
||||
nonexistent.invalid.example is neither an inventory file, a (list of) hosts…
|
||||
EXIT=0
|
||||
--- STDOUT --- (empty)
|
||||
```
|
||||
|
||||
Absence of a probe result must be a hard failure. It must never be read as
|
||||
"this host has no requirements to check", which is the shape the bug would take.
|
||||
The `###NOPY-FACTS###` sentinel exists for this: one line per host is *required*,
|
||||
and a missing line is an error.
|
||||
|
||||
### `server.Users` already covers the motivating case
|
||||
|
||||
It returns `shell`, `groups`, `home`, `uid`, `gid`, `comment` and `password` per
|
||||
user, keyed by name. `user:add` needs almost no custom fact code — see
|
||||
[Sharp edges](#sharp-edges) for why `password` is a problem.
|
||||
|
||||
## The proposal
|
||||
|
||||
Split the one idea into two manifest fields, because there are two different
|
||||
objects: what a cube can **report** about a host (owned by the cube responsible
|
||||
for that state) and what a cube **demands** (owned by the consumer).
|
||||
|
||||
A single `facts:` field cannot be both.
|
||||
|
||||
### `provides` — on the cube that owns the state
|
||||
|
||||
```js
|
||||
// cubes/user/add/manifest.mjs
|
||||
export default Manifest({
|
||||
id: 'user:add',
|
||||
provides: {
|
||||
/** What the probe returns. Validated on the way back in. */
|
||||
schema: z.object({
|
||||
exists: z.boolean(),
|
||||
shell: z.string().nullable(),
|
||||
groups: z.array(z.string()),
|
||||
}),
|
||||
/** Schema keys the probe needs in order to look anything up. */
|
||||
params: ['USER'],
|
||||
},
|
||||
schema: z.object({ /* … unchanged … */ }),
|
||||
});
|
||||
```
|
||||
|
||||
Plus a `facts.py` in the cube directory, discovered by convention exactly as
|
||||
`deploy.py` is.
|
||||
|
||||
### `requires` — on the consumer
|
||||
|
||||
```js
|
||||
requires: (vars) => [{
|
||||
cube: 'user:add',
|
||||
with: { USER: vars.DEPLOY_USER },
|
||||
expect: z.object({
|
||||
exists: z.literal(true),
|
||||
shell: z.literal('/usr/bin/fish'),
|
||||
groups: z.array(z.string()).refine((g) => g.includes('docker')),
|
||||
}),
|
||||
}],
|
||||
```
|
||||
|
||||
Three deliberate choices:
|
||||
|
||||
**A requirement names its provider.** If `requires` stated only a predicate, the
|
||||
engine would have to search the cube space for something that satisfies an
|
||||
arbitrary Zod schema. That is a planner, and a planner is a research project.
|
||||
Naming the cube keeps resolution linear and keeps the failure message legible.
|
||||
|
||||
**`requires` is `(vars) => …`, like `dependencies` already is.** A requirement
|
||||
almost always depends on the consumer's own variables — you cannot know *which*
|
||||
user to check for until the consumer has been asked.
|
||||
|
||||
**Zod is the predicate language.** It is already this repo's schema vocabulary,
|
||||
`z.literal` and `.refine()` cover the cases, and `z.treeifyError()` produces the
|
||||
failure report for free. The cost is that `.refine()` closures are not
|
||||
serialisable, so a requirement can never be written into a session — only its
|
||||
*result* can. See [Sharp edges](#sharp-edges).
|
||||
|
||||
### The engine's rule
|
||||
|
||||
This is the half of the design that "only run if requirements are fulfilled"
|
||||
leaves out. When a requirement is **not** met, there are three possible answers:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Abort** | Honest, and useless. On a bare host nothing is fulfilled, so every fresh deploy fails. |
|
||||
| **Skip the cube** | A silent no-op. Dangerous. |
|
||||
| **Remedy** | Run the named cube. |
|
||||
|
||||
It has to be *remedy* — and remedy means "run the cube that provides it", which
|
||||
is `dependencies` again. That is the actual insight here, and it is a much
|
||||
smaller change than a parallel subsystem:
|
||||
|
||||
> **You do not need a new mechanism. You need dependencies to become conditional.**
|
||||
|
||||
So:
|
||||
|
||||
- requirement **met** → the provider is *not* scheduled and its variables are
|
||||
*not* prompted for. This is where the prompt tax disappears.
|
||||
- requirement **unmet** → the provider is scheduled with `with` applied as
|
||||
overrides. `with` assigns at origin `param`, which already outranks every
|
||||
other origin (`default < env < session < prompt < param`), so no new
|
||||
precedence rule is needed.
|
||||
|
||||
## Why `facts.py` exposes a function, not a script
|
||||
|
||||
The obvious design is symmetry: `deploy.py` is a script, so `facts.py` is a
|
||||
script. Reject it.
|
||||
|
||||
A recursive resolution over a dependency tree issues one probe per (cube, host,
|
||||
params). At one pyinfra invocation each, that is one SSH connection each —
|
||||
seconds apiece, multiplied by the tree. Unaffordable.
|
||||
|
||||
But probes are **pure reads with no ordering constraints between them**, which
|
||||
means every probe for a given host can be gathered in a *single* pyinfra
|
||||
invocation: one generated driver script that imports each cube's `facts.py`,
|
||||
calls it with its params, and emits one JSON object per host. One connection per
|
||||
host per resolution round, instead of one per cube.
|
||||
|
||||
A standalone script can only be run alone. A function can be batched:
|
||||
|
||||
```python
|
||||
# cubes/user/add/facts.py
|
||||
from pyinfra.facts.server import Users
|
||||
|
||||
def gather(host, params):
|
||||
u = host.get_fact(Users).get(params["USER"])
|
||||
return {
|
||||
"exists": u is not None,
|
||||
"shell": (u or {}).get("shell"),
|
||||
"groups": (u or {}).get("groups", []),
|
||||
}
|
||||
```
|
||||
|
||||
Consequence: params arrive as one `--data NOPY_PROBES=<json>` blob rather than
|
||||
per-cube `--data KEY=…`, since a batched run carries several cubes' params at
|
||||
once. Probe results are memoised per (cube, host, params) in the `BuildContext`,
|
||||
the same shape as the existing `resolvedCubes` set.
|
||||
|
||||
## Execution model
|
||||
|
||||
Probes run during **resolution**, before anything has deployed. That has a
|
||||
consequence worth stating plainly rather than discovering later:
|
||||
|
||||
When the plan reads *"user absent → schedule `user:add` → then B"*, B's
|
||||
requirement was never verified. It was **assumed** met because a remedy was
|
||||
scheduled. Every tool in this space stops there.
|
||||
|
||||
Since the probe already exists, the loop can be closed for nearly nothing:
|
||||
|
||||
> **Re-run the provider's probe after it deploys, and fail loudly if the
|
||||
> requirement still is not met.**
|
||||
|
||||
`facts.py` then serves as a post-condition test as well as a precondition check.
|
||||
This is the single most valuable property of the design — it is strictly more
|
||||
than Ansible's `when:` offers — and it should be built in from the start rather
|
||||
than added as a later refinement.
|
||||
|
||||
Sketch of the resolution change in `BuildContext.resolveCube`, between steps 3
|
||||
(`before` hooks) and 4 (dependencies):
|
||||
|
||||
1. Collect `manifest.requires?.(currentVars)`.
|
||||
2. Batch every unmet-so-far probe for this host into one pyinfra run.
|
||||
3. Parse each result against the provider's `provides.schema`, then against the
|
||||
consumer's `expect`.
|
||||
4. For each failure, `resolveCube(req.cube, host, req.with)`.
|
||||
5. After that provider's deploy call executes, re-probe and assert.
|
||||
|
||||
Step 5 does not fit the current shape: `deployCalls` are all built first and
|
||||
executed later, so a post-condition needs the executor to call back into
|
||||
probing. That is the one structurally invasive part of this proposal.
|
||||
|
||||
## Sharp edges
|
||||
|
||||
**`--print-only` purity.** A dry run touches no host today. Probing breaks that
|
||||
outright. Needs an explicit answer — either a probe-free plan that renders
|
||||
requirements as unevaluated, or an opt-in flag. Silently connecting during what
|
||||
the user believes is a dry run is not acceptable.
|
||||
|
||||
**Secret leakage.** `server.Users` returns the **encrypted password hash** in
|
||||
its `password` field. A probe result that reaches history, a plan dump, or a log
|
||||
line is a credential leak. The existing `secrets` machinery is per-*variable*
|
||||
and does not apply — facts need their own redaction rule. Easy to miss, hard to
|
||||
take back.
|
||||
|
||||
**Sessions must never persist facts.** Facts are host state at a moment in time;
|
||||
a replay must re-probe, never restore. Recording them in *history* for
|
||||
diagnostics is fine and useful. This is also forced by `.refine()` being
|
||||
unserialisable.
|
||||
|
||||
**Sudo.** Many useful facts require `_sudo`, and a probe runs before the deploy
|
||||
has made any of its own auth decisions. Unresolved — worth a spike.
|
||||
|
||||
**Cycles.** `requires` introduces a second edge type into a graph that still has
|
||||
no cycle detection (see *Known drift* in `CLAUDE.md`). Conditional edges make
|
||||
"A requires B, B requires A under different params" considerably more reachable
|
||||
than the current unconditional ones do.
|
||||
|
||||
**Naming.** `provides` / `requires` over the original `facts:`, because the
|
||||
field names should say which side of the relationship they belong to.
|
||||
|
||||
## Measured vs. assumed
|
||||
|
||||
Measured against pyinfra 3.5.1, on this machine:
|
||||
|
||||
- `fact` subcommand writes JSON to stderr and exits 0 on fact failure
|
||||
- a deploy file's stdout is uncontaminated by pyinfra output
|
||||
- `--data` reaches a probe as `host.data.KEY`
|
||||
- a deploy file with zero operations succeeds
|
||||
- a custom `FactBase` in a sibling module resolves under `--chdir`
|
||||
- an exception in a deploy file exits 1
|
||||
- **an unreachable host exits 0 with empty stdout**
|
||||
|
||||
Assumed, not yet tested:
|
||||
|
||||
- batching several cubes' probes into one driver script works and is meaningfully
|
||||
cheaper than N invocations — this is the load-bearing cost assumption and
|
||||
should be the first thing spiked
|
||||
- `host.get_fact()` accepts `_sudo` from within a probe function
|
||||
- the post-condition re-probe can be threaded through `executeDeployCalls`
|
||||
without unpicking the build-then-execute split
|
||||
@@ -2,9 +2,14 @@
|
||||
|
||||
Nopy supports two session file formats: **JSON** and **MJS** (ES Module JavaScript).
|
||||
|
||||
The extension is what picks the loader, so a session file has to end in `.json`
|
||||
or `.mjs`; anything else is refused by name. The `.nopysession.*` names used
|
||||
throughout are the convention `listSessions()` looks for — `-s` and `-l` accept
|
||||
any path you give them.
|
||||
|
||||
## Supported Formats
|
||||
|
||||
### JSON Format (`.session.json`)
|
||||
### JSON Format (`.nopysession.json`)
|
||||
|
||||
Traditional JSON format for session files:
|
||||
|
||||
@@ -35,7 +40,7 @@ Traditional JSON format for session files:
|
||||
- Cannot use dynamic values or computation
|
||||
- No code reuse or imports
|
||||
|
||||
### MJS Format (`.session.mjs`) - **Recommended**
|
||||
### MJS Format (`.nopysession.mjs`) - **Recommended**
|
||||
|
||||
JavaScript module format with full ES Module support:
|
||||
|
||||
@@ -172,7 +177,7 @@ export const commonCubes = [
|
||||
];
|
||||
```
|
||||
|
||||
**my-session.session.mjs:**
|
||||
**my-session.nopysession.mjs:**
|
||||
```javascript
|
||||
import { productionHosts, commonCubes } from './common-config.mjs';
|
||||
|
||||
@@ -232,7 +237,7 @@ function generateSession(config) {
|
||||
};
|
||||
|
||||
const content = `export default ${JSON.stringify(session, null, 2)};`;
|
||||
fs.writeFileSync('generated.session.mjs', content);
|
||||
fs.writeFileSync('generated.nopysession.mjs', content);
|
||||
}
|
||||
|
||||
// Generate from external configuration
|
||||
@@ -255,10 +260,10 @@ Both formats are loaded the same way:
|
||||
import { loadSession } from '@bitsquare/nopy';
|
||||
|
||||
// Load JSON
|
||||
const jsonSession = await loadSession('./my-session.session.json');
|
||||
const jsonSession = await loadSession('./my-session.nopysession.json');
|
||||
|
||||
// Load MJS
|
||||
const mjsSession = await loadSession('./my-session.session.mjs');
|
||||
const mjsSession = await loadSession('./my-session.nopysession.mjs');
|
||||
```
|
||||
|
||||
The file extension determines which loader to use.
|
||||
@@ -267,7 +272,7 @@ The file extension determines which loader to use.
|
||||
|
||||
To convert an existing JSON session to MJS:
|
||||
|
||||
1. Rename the file from `.session.json` to `.session.mjs`
|
||||
1. Rename the file from `.nopysession.json` to `.nopysession.mjs`
|
||||
2. Add `export default` before the configuration object
|
||||
3. Remove quotes from property keys (optional)
|
||||
4. Add comments and dynamic values as needed
|
||||
@@ -304,11 +309,12 @@ Both formats must export/contain an object with this structure:
|
||||
|
||||
```typescript
|
||||
interface NopySession {
|
||||
version: string; // Session format version
|
||||
timestamp: string; // ISO timestamp
|
||||
cubes: CubeSession[]; // Array of cube configurations
|
||||
hosts: string[]; // Target hosts
|
||||
auth: AuthSession; // Authentication configuration
|
||||
cubes: CubeSession[]; // Array of cube configurations — required
|
||||
auth: AuthSession; // Authentication configuration — required
|
||||
version?: string; // Session format version, currently "1.0.0"
|
||||
timestamp?: string; // ISO timestamp
|
||||
name?: string; // One-line description
|
||||
hosts?: string[]; // Target hosts
|
||||
env?: Record<string, any>; // Global environment variables
|
||||
}
|
||||
|
||||
@@ -323,6 +329,17 @@ interface AuthSession {
|
||||
}
|
||||
```
|
||||
|
||||
Only `cubes` and `auth` are demanded of a session being *read* — the loader
|
||||
requires what it cannot work without and nothing else, so the sessions in these
|
||||
examples are all valid, and one written before `version` existed still loads. A
|
||||
session nopy *writes* always carries `version`, `timestamp` and `name`; a
|
||||
`version` this build does not recognise produces a warning on stderr and loads
|
||||
anyway.
|
||||
|
||||
`method: 'ssh'` is the third value and the one no prompt produces: it means the
|
||||
connector handles authentication and nopy supplies no credential. Every
|
||||
`@vagrant/` and `@docker/` host gets it.
|
||||
|
||||
A session nopy *writes* holds, per cube, every value that cube ran with — what
|
||||
was typed, what came from `.nopyrc.json`, what a dependency supplied, and what
|
||||
fell through to the schema's `.default()`. Two things are deliberately absent and
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
# Vagrant
|
||||
|
||||
`vagrant ssh-config` to find the SSH port of the machine
|
||||
`vagrant status --machine-readable` will be executed by pyinfra to get information about available VMs
|
||||
A Vagrant box is the cheapest way to run a cube against a real machine you can
|
||||
throw away afterwards.
|
||||
|
||||
## The Vagrantfile
|
||||
|
||||
```ruby
|
||||
|
||||
@@ -19,3 +21,34 @@ Vagrant.configure("2") do |config|
|
||||
end
|
||||
|
||||
```
|
||||
|
||||
## Naming the machine to nopy
|
||||
|
||||
The host string is `@vagrant/<name>`, where `<name>` is what `config.vm.define`
|
||||
declared — `nopytestvm` above. It is pyinfra's connector syntax, not nopy's, and
|
||||
it is the same shape as `@docker/<container-or-image>`.
|
||||
|
||||
Two ways to get there. Either pick `vagrant` in the host prompt and answer the
|
||||
follow-up with the machine name, which is what builds the string for you, or put
|
||||
it in `.nopyrc.json` so it appears in the list directly:
|
||||
|
||||
```json
|
||||
{
|
||||
"hosts": ["@vagrant/nopytestvm"],
|
||||
"cubePackages": ["@bitsquare/nopy-cubes-core"]
|
||||
}
|
||||
```
|
||||
|
||||
pyinfra runs `vagrant status --machine-readable` to find the available machines
|
||||
and `vagrant ssh-config` for the SSH port, so `vagrant` has to be on `PATH` and
|
||||
the box has to be `up` before a deploy.
|
||||
|
||||
## Cleaning up
|
||||
|
||||
```sh
|
||||
vagrant halt # stop it, keep the disk
|
||||
vagrant destroy -f # delete it — the next `vagrant up` is a fresh box
|
||||
```
|
||||
|
||||
`destroy` is the one to use between test runs of a cube that is not idempotent:
|
||||
re-running against a half-configured box tests something other than the cube.
|
||||
|
||||
Reference in New Issue
Block a user