ansiblings/packages/nopy/docs/REFACTORING.md

Nopy Refactoring Plan

This document tracks the major refactoring of the nopy package.

Refactoring Items

1. Remove parallel execution

• Status: ✅ Completed
• Goal: Remove all logic supporting parallel execution of cubes to simplify the execution flow and improve reliability.
• Rationale: The feature never shipped. Concurrent pyinfra processes interleave their output, which made deployment logs unreadable — a cost that outweighed the wall-clock saving. Do not reintroduce it without first solving per-cube output buffering.
• Context:
  • Parallelism removed from NopyConfig, NopyOptions, and executeDeployCalls.
  • buildExecutionStages deleted.
  • CLI flags --parallel and --concurrency removed.
  • Documentation caught up later: README.md, docs/API.md, and docs/HOOKS.md had all continued to describe the feature as if it existed.
• Proposed Solution: (Done)

2. Rework cube building process & Dependency Resolution

• Status: ✅ Completed
• Goal: Allow dependencies to be defined as a function of the collected variables.
• New Signature: dependencies?: (variables: CubeVariables) => DependencySpec[]
• Architectural Change: Implement a clean, step-based resolution mechanism using a BuildContext.
• Context:
  • Introduced BuildContext in cubes/dependencies.ts which handles recursive resolution, variable collection, and hook execution.
  • Resolution is now dynamic: variables are collected for a cube before its dependencies are resolved.
• Proposed Solution: (Done)

3. Remove env property from cube Manifest

• Status: ✅ Completed
• Goal: Remove the env property from the cube manifest.
• Context:
  • env removed from Manifest and Env types.
  • Responsibility for defaults shifted entirely to Zod schema defaults and getDefaults().
• Proposed Solution: (Done)

4. Redesign Manifest and Cube types

• Status: ✅ Completed
• Goal: Transition from Env -> Manifest -> Cube inheritance to a cleaner Manifest (specification) and Cube (runtime) separation.
• Context:
  • Manifest is now a clean interface with a factory namespace.
  • Cube is a class encapsulating a Manifest and runtime info (dir, deployScript).
• Proposed Solution: (Done)

5. Make --use-defaults operational

• Status: ✅ Completed
• Goal: Turn -D from a flag that was parsed and threaded through three layers but never read into a working non-interactive mode.
• Rationale: Unattended runs — CI, or provisioning a fresh box from a checked-in .nopyrc.json — are the reason the flag exists. It prompted anyway.
• Context:
  • BuildContext.resolveCube branches on options.useDefaults and skips VariableAssignment.
  • Variables.get() merge order corrected to defaults → global env → prompts → params. env used to lose to the schema default, which left a non-interactive run with no way to be configured at all.
  • Replayed session values moved from the defaults scope to prompts, so they keep outranking env now that env sits higher.
  • Cube.getDefaults() no longer discards every default when one field lacks .default(); it falls back to a per-field read.
  • Cube.requiredKeys() added, and a -D run fails naming the unfillable variables instead of deploying a cube with them absent from --data.
  • VariableAssignment offers every schema key, not only the ones carrying a default, and shows the value the run would actually use as the initial.
• Proposed Solution: (Done)

6. Make variable assignment a first-class concept

• Status: ✅ Completed
• Goal: Give a variable an identity and a provenance, instead of inferring both from which bag it happened to sit in.
• Rationale: Item 5 left precedence encoded as the field order of an object literal inside Variables.get() — defaults, then global, then prompts, then params. Nothing named the ranking, nothing could be asked where a value came from, and a replay had to be smuggled into the prompts bag because there was no origin that meant "recorded". Every question that followed — what should a session record, which values are safe to print — needed provenance to answer.
• Context:
  • Assignment { value, origin } and an Origin ranked default(0) < env(1) < session(2) < prompt(3) < param(4). Precedence is now data, not the order lines appear in.
  • Variable is a class over an assignment list. assignments is the true history, newest first and never reordered; ordered is a stable sort of it by origin rank, and value/origin read the head of that. Stability is what makes the two views coexist: same-origin ties keep the newest in front while the value it displaced stays visible.
  • The global bag is gone. Config env is seeded per cube as a real assignment at origin env, so variables.get('global') — a cube id that was never a cube — is no longer a thing.
  • Replay assigns at origin session, which outranks env and default on its own. The prompts-bag workaround is deleted.
  • A session records Variables.persistable() — every effective value, not just prompted ones. A -D run used to record nothing and replay by re-deriving from whatever the defaults said at replay time.
• Trade-off accepted: recorded values now outrank the current .nopyrc.json env and the current schema defaults, so editing either no longer leaks into an existing session's replay. That is the point of a snapshot, but it does mean picking up a new default requires re-recording.
• Proposed Solution: (Done)

7. Manifest-declared secrets

• Status: ✅ Completed
• Goal: Let a manifest say which schema keys hold sensitive values, and act on it.
• Rationale: Item 6 made sessions record everything, which forced the question of what must not be recorded. The codebase already had an answer of sorts — outputExecutionPlan masked any variable whose name contained "password" — that missed TOKEN, PSK and AUTH_KEY, and was defeated anyway by the unmasked command printed one line above it.
• Context:
  • Manifest.secrets?: string[], validated at load: an entry that is not a key of schema is a manifest error and aborts the run, so a typo cannot silently leave a value unprotected.
  • Deliberately a plain array, not zod metadata. .meta() and .describe() store into z.globalRegistry, which is per-copy — a manifest built by a different zod copy would look up empty. Fail-open is fine for a missing prompt label and unacceptable for a secret marker.
  • maskCommand() replaces declared --data values and the SSH --password in the command string itself, and is wired into --print-only, the dry-run plan and the debug log. The nopy logger runs at lowestLevel: 'debug', so that last one was printing credentials on every run.
  • Secrets are excluded from persistable(), so a replay has a gap where one used to be. fillSessionGaps prompts for requiredKeys() ∪ secrets; under -D it fails naming them, consistent with item 5's fail-fast.
• Scope limit: secrets keeps a value out of what nopy writes. The value is still on pyinfra's command line (visible in ps), still echoed by the variable form, and a .default() is still plain text in the manifest. Documented rather than fixed — the first is inherent to pyinfra's interface.
• Bug fixed along the way: cubes/user/add generated a random password as its schema .default(). Because the key had a default it was never in requiredKeys(), and because a generated default is re-evaluated on every read, an unattended run created an account with a credential nobody had seen and a replay created a different one again. It is now the literal changeme.
• Proposed Solution: (Done)