initial transfer

This commit is contained in:
Benjamin Diedrichsen
2026-07-27 13:09:00 +02:00
parent 9f25d48dc2
commit 736c01216a
191 changed files with 17622 additions and 136 deletions
+655
View File
@@ -0,0 +1,655 @@
# Nopy API Reference
This document describes the public API for the nopy package.
---
## Table of Contents
- [Main Module](#main-module)
- [Cubes Module](#cubes-module)
- [Executor Module](#executor-module)
- [Builder Module](#builder-module)
- [Workflow Module](#workflow-module)
- [Session Module](#session-module)
- [Config Module](#config-module)
- [Prompts Module](#prompts-module)
---
## Main Module
### `nopy(options?)`
Main entry point for nopy deployments.
```typescript
import { nopy } from '@bitstack/nopy';
const result = await nopy({
useDefaults: false,
dryRun: true,
});
```
**Parameters:**
| Name | Type | Default | Description |
|------|------|---------|-------------|
| `useDefaults` | `boolean` | `false` | Skip variable prompts, use defaults |
| `useAuthKey` | `boolean` | `false` | Force SSH key authentication |
| `saveSession` | `string` | - | Path to save session file |
| `loadSession` | `string` | - | Path to load session for replay |
| `dryRun` | `boolean` | `false` | Show execution plan without running |
| `parallel` | `boolean` | `false` | Execute independent cubes in parallel |
| `continueOnError` | `boolean` | `false` | Continue after failures |
| `jsonOutput` | `boolean` | `false` | Output results as JSON |
**Returns:** `Promise<NopyResult | undefined>`
```typescript
interface NopyResult {
success: boolean;
results: ExecutionResult[];
summary: {
total: number;
successful: number;
failed: number;
totalDuration: number;
};
}
```
---
## Cubes Module
The cubes module provides types and functions for working with deployment units.
### Types
#### `Cube<Schema>`
A fully loaded cube with filesystem location.
```typescript
interface Cube<Schema extends z.AnyZodObject = z.AnyZodObject> {
key: string; // Unique identifier
name: string; // Human-readable name
dir: string; // Absolute path to cube directory
dependencies: string[];
schema: Schema;
defaults: () => z.infer<Schema>;
before: Hook<Schema>[];
after: Hook<Schema>[];
}
```
#### `Manifest<Schema>`
Cube manifest (used in `*.manifest.mjs` files).
```typescript
interface Manifest<Schema extends z.AnyZodObject = z.AnyZodObject> {
name: string;
key: string;
dependencies: string[];
schema: Schema;
defaults: () => z.infer<Schema>;
before: Hook<Schema>[];
after: Hook<Schema>[];
}
```
#### `Hook<Schema>`
Hook function for before/after cube execution. See [Cube Hooks](HOOKS.md) for more details.
```typescript
type Hook<Schema extends z.AnyZodObject> = (
ctx: HookContext,
params: z.infer<Schema>
) => void | Promise<void>;
interface HookContext {
/**
* Schedules another cube for execution.
* @param key - The unique identifier or path of the cube.
* @param params - Variables to pass to the cube.
*/
exec: (key: string, params: CubeVariables) => Promise<void> | void;
}
```
### Functions
#### `loadCubes()`
Loads all cubes from discovered cube directories.
```typescript
const { cubes, errors } = await loadCubes();
```
**Returns:** `Promise<LoadResult>`
```typescript
interface LoadResult {
cubes: Record<string, Cube>;
errors: string[];
}
```
#### `resolveDependencies(cubes, selectedCubeNames)`
Resolves all transitive dependencies for selected cubes.
```typescript
const order = resolveDependencies(cubes, ['apt-all']);
// Returns: ['apt:essentials', 'apt-more', 'apt-all']
```
**Parameters:**
| Name | Type | Description |
|------|------|-------------|
| `cubes` | `Record<string, Cube>` | Map of all available cubes |
| `selectedCubeNames` | `string[]` | Cubes to resolve |
**Returns:** `string[]` - Cube names in execution order
**Throws:** `Error` if cube not found or circular dependency detected
#### `buildExecutionStages(cubes, selectedCubeNames)`
Groups cubes into stages for parallel execution.
```typescript
const stages = buildExecutionStages(cubes, ['apt-all', 'docker']);
// Returns: [['apt:essentials'], ['apt-more', 'docker'], ['apt-all']]
```
**Returns:** `string[][]` - Array of stages
#### `createManifest(options)`
Factory function for creating cube manifests.
```typescript
export default createManifest({
name: 'My Cube',
dependencies: () => [['apt:essentials']],
schema: z.object({
VERSION: z.string().default('1.0'),
}),
});
```
#### `uniqid(length?)`
Generates a random alphanumeric string.
```typescript
const id = uniqid(); // 'Kx7Pm'
const long = uniqid(10); // 'Kx7PmQr2Yw'
```
---
## Executor Module
Handles pyinfra command execution.
### Types
#### `DeployCall`
A deployment command ready for execution.
```typescript
interface DeployCall {
cube: string;
host: string;
cwd: string;
command: string[];
env: Record<string, unknown>;
dependencies: string[];
}
```
#### `ExecutionResult`
Result of executing a deployment command.
```typescript
interface ExecutionResult {
cube: string;
host: string;
success: boolean;
duration: number;
stdout?: string;
stderr?: string;
error?: Error;
}
```
#### `ExecutionOptions`
Options for deployment execution.
```typescript
interface ExecutionOptions {
parallel?: boolean;
concurrency?: number;
continueOnError?: boolean;
dryRun?: boolean;
onProgress?: (result: ExecutionResult, completed: number, total: number) => void;
onStart?: (cube: string, host: string) => void;
}
```
### Functions
#### `executeDeployCalls(calls, options?)`
Executes an array of deployment calls.
```typescript
const results = await executeDeployCalls(calls, {
parallel: true,
concurrency: 4,
onProgress: (result, completed, total) => {
console.log(`${completed}/${total}`);
},
});
```
#### `outputExecutionPlan(calls, asJson?)`
Outputs the execution plan without running.
```typescript
outputExecutionPlan(deployCalls); // Text output
outputExecutionPlan(deployCalls, true); // JSON output
```
#### `summarizeResults(results)`
Generates a summary of execution results.
```typescript
const summary = summarizeResults(results);
// {
// total: 5,
// successful: 4,
// failed: 1,
// totalDuration: 12345,
// failures: [{ cube: 'docker', ... }]
// }
```
---
## Builder Module
Constructs deployment commands.
### `buildDeployCalls(cubeNames, hosts, context)`
Builds deployment calls for all cubes and hosts.
```typescript
const result = await buildDeployCalls(
['apt:essentials', 'apt-more'],
['@docker/test'],
{
cubes,
session,
config,
authMethod: 'ssh-key',
useDefaults: true,
isSessionReplay: false,
}
);
```
**Returns:** `Promise<BuildResult>`
```typescript
interface BuildResult {
deployCalls: DeployCall[];
cubeSessions: CubeSession[];
sessionEnv: Record<string, unknown>;
}
```
---
## Workflow Module
Manages interactive and replay workflows.
### `runWorkflow(sessionPath, cubes, config, options?)`
Runs the appropriate workflow based on options.
```typescript
const result = await runWorkflow(
undefined, // null for interactive, path for replay
cubes,
config,
{ useDefaults: false }
);
```
**Returns:** `Promise<WorkflowResult>`
```typescript
interface WorkflowResult {
session: NopySession;
cubesWithDependencies: string[];
authMethod: string;
username?: string;
password?: string;
isReplay: boolean;
}
```
### `runInteractiveWorkflow(cubes, config, options?)`
Runs the interactive cube selection workflow.
### `runReplayWorkflow(sessionPath, cubes, config)`
Runs a replay from a saved session file.
---
## Session Module
Manages session save/load operations.
### Types
#### `NopySession`
Complete session configuration.
```typescript
interface NopySession {
name?: string;
cubes: CubeSession[];
hosts?: string[];
auth: AuthSession;
env?: SessionVariables;
}
```
#### `CubeSession`
Configuration for a single cube.
```typescript
interface CubeSession {
key: string;
variables: SessionVariables;
}
```
#### `AuthSession`
Authentication configuration.
```typescript
interface AuthSession {
method: 'ssh-key' | 'password' | 'ssh';
username?: string;
}
```
### Functions
#### `saveSession(session, filePath)`
Saves a session to a JSON file.
```typescript
saveSession(session, './my-deployment.nopysession.json');
```
#### `loadSession(filePath)`
Loads a session from a JSON or MJS file.
```typescript
const session = await loadSession('./deployment.json');
const session = await loadSession('./deployment.mjs');
```
#### `createSession(params)`
Creates a session object from runtime data.
```typescript
const session = createSession({
cubes: [{ key: 'apt:essentials', variables: {} }],
hosts: ['localhost'],
auth: { method: 'ssh-key' },
});
```
#### `listSessions(dirPath?)`
Lists all session files in a directory.
```typescript
const sessions = listSessions('./sessions');
// ['./sessions/deploy.session.json', './sessions/test.session.mjs']
```
---
## Config Module
Manages nopy configuration.
### Types
#### `NopyConfig`
Configuration file structure.
```typescript
interface NopyConfig {
hosts: string[];
cubeDirs: string[];
env: EnvConfig;
log?: LogConfig;
}
```
#### `LogConfig`
Logging configuration.
```typescript
interface LogConfig {
verbosity?: 'silent' | 'info' | 'verbose' | 'trace';
debug?: boolean;
}
```
### Functions
#### `loadConfig()`
Loads configuration from `.nopyrc.json`.
```typescript
const config = loadConfig();
```
Search order:
1. `./nopyrc.json` (local)
2. `~/.nopyrc.json` (home)
#### `saveConfig(data, local?)`
Saves configuration to a file.
```typescript
saveConfig({ hosts: ['server.local'] }); // Local
saveConfig({ hosts: ['server.local'] }, false); // Home
```
#### `logConfigToFlags(logConfig?)`
Converts log config to pyinfra flags.
```typescript
logConfigToFlags({ verbosity: 'verbose', debug: true });
// ['-vv', '--debug']
```
---
## Prompts Module
Interactive prompts for user input.
### `CubeSelection(cubes)`
Prompts user to select cubes to execute.
```typescript
const { selectedCubes } = await CubeSelection(cubes);
```
### `HostSelection(hosts)`
Prompts user to select a target host.
```typescript
const host = await HostSelection(['server1', 'server2']);
```
### `AuthSelection(useAuthKey?)`
Prompts user to select authentication method.
```typescript
const { authMethod, username, password } = await AuthSelection();
```
### `VariableAssignment(cube, env)`
Prompts user to customize cube variables.
```typescript
const vars = await VariableAssignment(cube, { existing: 'value' });
```
### `PasswordSelection(username)`
Prompts for password input.
```typescript
const password = await PasswordSelection('admin');
```
---
## CLI Usage
```bash
# Interactive deployment
nopy install
# With defaults (no prompts)
nopy install -D
# SSH key auth
nopy install -K
# Save session
nopy install -s ./my-session.json
# Replay session
nopy install -l ./my-session.json
# Dry run
nopy install -n
# Parallel execution
nopy install -p
# JSON output
nopy install -j
# Continue on error
nopy install -c
```
---
## Creating a Cube
### File Structure
```
cubes/
└── my-cube/
├── my-cube.manifest.mjs
└── my-cube.deploy.py
```
### Manifest Example
```javascript
// my-cube.manifest.mjs
import { createManifest } from '@bitstack/nopy';
import { z } from 'zod';
export default createManifest({
name: 'My Cube',
dependencies: () => [['apt:essentials']],
schema: z.object({
VERSION: z.string().default('1.0').describe('Version to install'),
ENABLE_FEATURE: z.boolean().default(false),
}),
before: [
(ctx, params) => {
console.log('Before my-cube');
},
],
after: [
(ctx, params) => {
console.log('After my-cube');
},
],
});
```
### Deploy Script Example
```python
# my-cube.deploy.py
from pyinfra import host
from pyinfra.operations import apt, server
VERSION = host.data.get('VERSION', '1.0')
ENABLE_FEATURE = host.data.get('ENABLE_FEATURE', False)
apt.packages(
name='Install my-package',
packages=[f'my-package={VERSION}'],
update=True,
)
if ENABLE_FEATURE:
server.shell(
name='Enable feature',
commands=['my-package --enable-feature'],
)
```
+90
View File
@@ -0,0 +1,90 @@
# Nopy Cube Hooks
Hooks provide a way to orchestrate deployments dynamically during the build process. They allow a cube to trigger the execution of other cubes based on its configuration or the environment.
## Overview
A cube manifest can define `before` and `after` hooks. These hooks are executed when the deployment plan is being built.
- **`before` hooks**: Executed *before* the current cube is added to the deployment sequence.
- **`after` hooks**: Executed *after* the current cube is added to the deployment sequence.
## Specification
Hooks are defined as an array of functions in the cube manifest.
```javascript
import { z } from 'zod';
import { cubes } from '@bitstack/nopy';
export default cubes.Manifest({
name: 'my-cube',
schema: z.object({
SETUP_DB: z.boolean().default(false),
}),
before: [
async ({ exec }, params) => {
if (params.SETUP_DB) {
// This will run BEFORE my-cube
await exec('db:setup', { TYPE: 'postgres' });
}
}
],
after: [
({ exec }, params) => {
// This will run AFTER my-cube
console.log('Finished setting up my-cube');
}
]
});
```
### Hook Function Signature
Each hook function receives two arguments:
1. **`context`**: An object containing:
- `exec(cubeKey: string, params: Record<string, any>)`: A function to schedule another cube for execution.
2. **`params`**: The final, validated variables for the current cube (including defaults and user-provided values).
Hooks can be synchronous or asynchronous (returning a `Promise`).
## Mechanics
### Sequential Execution
In sequential execution mode (the default), cubes added via hooks will follow the order in which they were pushed to the deployment plan:
1. Cubes from `before` hooks.
2. The current cube itself.
3. Cubes from `after` hooks.
### Parallel Execution
In parallel execution mode, cubes added via hooks **do not automatically inherit dependencies**.
If a `before` hook calls `exec('setup-cube')`, it ensures that `setup-cube` is placed earlier in the deployment plan, but for parallel execution, you should still ensure that dependencies are correctly specified if one cube relies on another's completion.
### Variable Passing
When you call `exec(cubeKey, params)` within a hook:
1. The `params` provided are merged with the current environment variables.
2. These variables are passed to the target cube, preventing it from prompting the user for those same variables.
### Use Cases
- **Conditional Setup**: Running a setup cube only if a specific variable is set.
- **Environment Preparation**: Ensuring a user exists or a directory is created before the main cube runs.
- **Cleanup/Notification**: Running a task after a cube deployment finishes.
## Comparison with Dependencies
| Feature | Dependencies | Hooks |
| :--- | :--- | :--- |
| **Declaration** | Static (`dependencies: () => [['id']]`) | Dynamic (`before: [...]`) |
| **Execution Order** | Guaranteed before dependent | `before` (before) or `after` (after) |
| **Variable Passing** | Inherited from env | Explicitly passed via `exec()` |
| **Conditionality** | Always run | Can be conditional based on logic |
Use **dependencies** for static requirements and **hooks** for dynamic orchestration and explicit parameter passing.
+323
View File
@@ -0,0 +1,323 @@
# Nopy Session Format
Nopy supports two session file formats: **JSON** and **MJS** (ES Module JavaScript).
## Supported Formats
### JSON Format (`.session.json`)
Traditional JSON format for session files:
```json
{
"version": "1.0.0",
"timestamp": "2025-10-15T00:00:00.000Z",
"cubes": [
{
"key": "runtime:nodevm",
"variables": {
"VERSION": "22",
"USER": "myuser"
}
}
],
"hosts": ["@ssh/myhost.local"],
"auth": {
"method": "password",
"username": "admin"
},
"env": {}
}
```
**Limitations:**
- No comments allowed (pure JSON)
- Cannot use dynamic values or computation
- No code reuse or imports
### MJS Format (`.session.mjs`) - **Recommended**
JavaScript module format with full ES Module support:
```javascript
// Nopy Session Configuration
// Comments are fully supported!
// You can import values from other files
import { commonHosts } from './common-config.mjs';
// You can use dynamic values
const timestamp = new Date().toISOString();
const nodeVersion = process.env.NODE_VERSION || "22";
export default {
version: "1.0.0",
timestamp,
cubes: [
// Inline comments for each cube
{
key: "runtime:nodevm",
variables: {
VERSION: nodeVersion, // Dynamic value
USER: "myuser",
ALIAS: "nodelts",
GLOBAL_PACKAGES: "pm2 yarn"
}
},
// Add more cubes...
],
hosts: commonHosts, // Imported from another file
auth: {
method: "password",
username: "admin"
},
env: {
NODE_ENV: process.env.NODE_ENV || "production"
}
};
```
**Advantages:**
-**Comments** - Document your configuration inline
-**Dynamic values** - Use environment variables, compute values
-**Code reuse** - Import common configurations from other files
-**Parameterization** - Easily parameterize sessions from external tools
-**Type safety** - Use JSDoc or TypeScript for validation
-**Computation** - Calculate values, filter arrays, etc.
## Advanced MJS Examples
### Using Environment Variables
```javascript
export default {
version: "1.0.0",
timestamp: new Date().toISOString(),
cubes: [
{
key: "typestack-install",
variables: {
REPO: process.env.GIT_REPO || "git@github.com:org/repo.git",
USER: process.env.DEPLOY_USER || "admin",
APP: process.env.APP_NAME || "myapp",
ENV: process.env.NODE_ENV || "production"
}
}
],
hosts: [process.env.TARGET_HOST || "@ssh/localhost"],
auth: {
method: "password",
username: process.env.SSH_USER || "admin"
}
};
```
### Conditional Cube Inclusion
```javascript
const isDevelopment = process.env.NODE_ENV === 'development';
export default {
version: "1.0.0",
timestamp: new Date().toISOString(),
cubes: [
{
key: "runtime:docker",
variables: { DISTRO: "debian" }
},
// Only include in development
...(isDevelopment ? [{
key: "debug-tools",
variables: { INSTALL_GDB: true }
}] : [])
],
hosts: ["@ssh/myhost.local"],
auth: { method: "ssh" }
};
```
### Importing Common Configuration
**common-config.mjs:**
```javascript
export const productionHosts = [
"@ssh/prod-server-1.local",
"@ssh/prod-server-2.local"
];
export const stagingHosts = [
"@ssh/staging.local"
];
export const commonCubes = [
{
key: "apt:essentials",
variables: { UPDATE: true }
},
{
key: "runtime:docker",
variables: { DISTRO: "debian" }
}
];
```
**my-session.session.mjs:**
```javascript
import { productionHosts, commonCubes } from './common-config.mjs';
export default {
version: "1.0.0",
timestamp: new Date().toISOString(),
cubes: [
...commonCubes, // Include common cubes
{
key: "typestack-install",
variables: {
REPO: "git@github.com:myorg/myapp.git",
USER: "appuser",
APP: "myapp"
}
}
],
hosts: productionHosts, // Use imported hosts
auth: {
method: "password",
username: "admin"
}
};
```
### Programmatic Generation
You can even generate sessions programmatically from other tools:
**generate-session.mjs:**
```javascript
import fs from 'fs';
function generateSession(config) {
const cubes = config.services.map(service => ({
key: "typestack-install",
variables: {
REPO: service.repo,
USER: config.user,
APP: service.name,
ENV: config.environment
}
}));
const session = {
version: "1.0.0",
timestamp: new Date().toISOString(),
cubes,
hosts: config.hosts,
auth: {
method: "password",
username: config.user
}
};
const content = `export default ${JSON.stringify(session, null, 2)};`;
fs.writeFileSync('generated.session.mjs', content);
}
// Generate from external configuration
generateSession({
user: "deploy",
environment: "production",
services: [
{ name: "api", repo: "git@github.com:org/api.git" },
{ name: "web", repo: "git@github.com:org/web.git" }
],
hosts: ["@ssh/prod.local"]
});
```
## Loading Sessions
Both formats are loaded the same way:
```javascript
import { loadSession } from '@bitstack/nopy';
// Load JSON
const jsonSession = await loadSession('./my-session.session.json');
// Load MJS
const mjsSession = await loadSession('./my-session.session.mjs');
```
The file extension determines which loader to use.
## Migration from JSON to MJS
To convert an existing JSON session to MJS:
1. Rename the file from `.session.json` to `.session.mjs`
2. Add `export default` before the configuration object
3. Remove quotes from property keys (optional)
4. Add comments and dynamic values as needed
**Before (JSON):**
```json
{
"version": "1.0.0",
"cubes": [...]
}
```
**After (MJS):**
```javascript
export default {
version: "1.0.0",
cubes: [...]
};
```
## Best Practices
1. **Use MJS for new sessions** - Take advantage of comments and flexibility
2. **Document your cubes** - Add comments explaining what each cube does
3. **Use environment variables** - Make sessions reusable across environments
4. **Extract common config** - Share configuration across multiple sessions
5. **Version control** - Both formats work well with git
6. **Validate at runtime** - The loader validates the structure regardless of format
## Session Schema
Both formats must export/contain an object with this structure:
```typescript
interface NopySession {
version: string; // Session format version
timestamp: string; // ISO timestamp
cubes: CubeSession[]; // Array of cube configurations
hosts: string[]; // Target hosts
auth: AuthSession; // Authentication configuration
env?: Record<string, any>; // Global environment variables
}
interface CubeSession {
key: string; // Cube identifier
variables: Record<string, any>; // Cube-specific variables
}
interface AuthSession {
method: 'ssh-key' | 'password' | 'ssh';
username?: string;
}
```