BitSquare/ansiblings
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

[refactor] moving cubes into own package"
Publish snapshot / snapshot (push) Successful in 1m2s
Details

Browse Source 
[fix] default parameter run records parameters in session for replay[fix] remove default parameters for several cubes
This commit is contained in:
Benjamin Diedrichsen
2026-07-28 12:18:10 +02:00
parent ac050c4459
commit 6ecb2c366f
130 changed files with 3386 additions and 520 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/cubes-core/LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 bitsquare
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
packages/cubes-core/README.md
+59
View File
@@ -0,0 +1,59 @@
# @bitsquare/cubes-core
The core cube bundle for [nopy](https://www.npmjs.com/package/@bitsquare/nopy):
base packages, users, SSH, firewalling, networking, web serving and runtimes.
## Install
```sh
pnpm add -D @bitsquare/cubes-core
```
Then name it in `.nopyrc.json`:
```json
{
"hosts": ["web-1"],
"cubePackages": ["@bitsquare/cubes-core"]
}
```
`nopy` resolves the package from the directory of the config file that named it,
reads `nopy.cubes` out of its `package.json`, and scans those directories exactly
as it scans a `cubeDirs` entry. Nothing has to be linked or copied.
## What is in it
| Area | Cube ids |
| ---------- | ------------------------------------------------------------------- |
| admin | `admin:cockpit`, `admin:hostname`, `admin:locale` |
| packages | `apt:essentials`, `apt:install` |
| hardening | `armor:fail2ban`, `armor:ssh`, `armor:ufw` |
| web | `caddy`, `caddy:spa` |
| source | `git:clone` |
| networking | `net:tailscale`, `net:wifi:access-point`, `net:wifi:connection` |
| runtimes | `runtime:docker`, `runtime:nodevm` |
| services | `service:autostart` |
| ssh | `ssh:authorize`, `ssh:keygen`, `ssh:keyman` |
| users | `user:add`, `user:edit` |
Run `nopy` and pick from the list, or `nopy -P` to print the pyinfra commands
without executing them. Each cube directory has its own `README.md`.
## Cube ids are global
An id such as `apt:essentials` is claimed repo-wide, not per bundle: two cubes
with the same id — whichever sources they came from — abort the run with an
error naming both. Prefix your own cubes distinctly if you also point
`cubeDirs` at a local tree.
## The bundle is read-only
Under pnpm the installed files are hardlinked into the global store, so a cube
that writes next to its own `deploy.py` corrupts that store for every project on
the machine. Cubes here write to `/tmp` or to the remote host, never to their
own directory.
## License
MIT
packages/cubes-core/cubes/admin/cockpit/README.md
+76
View File
@@ -0,0 +1,76 @@
# cockpit
**Install Cockpit web-based server management interface**
## Purpose
This cube installs Cockpit, a powerful web-based interface for managing Linux servers, making system administration accessible through your browser.
## What is Cockpit?
Cockpit is a modern, interactive server admin interface that runs in your web browser. It provides:
- **Real-time monitoring**: CPU, memory, disk, and network usage graphs
- **Container management**: View and manage Docker containers
- **Service management**: Start, stop, and manage systemd services
- **Storage administration**: Manage disks, RAID, and filesystems
- **Network configuration**: Configure network interfaces and firewall
- **Terminal access**: Built-in terminal for command-line access
- **User management**: Create and manage user accounts
- **Software updates**: View and apply system updates
Think of it as a control panel for your Linux server - all accessible from any web browser.
## What This Cube Does
1. Installs the `cockpit` package
2. Installs `sscg` (Simple Signed Certificate Generator) for HTTPS support
3. Starts the Cockpit service
4. Makes Cockpit accessible on port 9090
## Configuration
This cube currently has no configurable parameters.
## Dependencies
None - this cube can run standalone.
## Post-Installation
Access Cockpit by navigating to:
```
https://your-server-ip:9090
```
Login with any valid system user account (e.g., root or a user created with the `user-add` cube).
### Security Notes
- Cockpit uses HTTPS by default (self-signed certificate)
- Your browser will show a security warning on first access (expected with self-signed certs)
- Consider using UFW to restrict access: `sudo ufw allow from YOUR_IP to any port 9090`
- Disable Cockpit when not in use: `sudo systemctl stop cockpit.socket`
## Common Use Cases
- Monitor server performance in real-time
- Manage Docker containers without command-line
- View system logs and troubleshoot issues
- Configure network settings
- Apply system updates
- Manage storage and filesystems
## Managing Cockpit
Start/stop Cockpit:
```bash
sudo systemctl start cockpit.socket
sudo systemctl stop cockpit.socket
sudo systemctl status cockpit.socket
```
Disable Cockpit from starting on boot:
```bash
sudo systemctl disable cockpit.socket
```
packages/cubes-core/cubes/admin/cockpit/deploy.py
+13
View File
@@ -0,0 +1,13 @@
from pyinfra import host
from pyinfra.operations import server, apt
apt.packages(
packages=[ "sscg cockpit"],
present=True,
_sudo=True
)
server.service(
'cockpit',
running=True,
)
packages/cubes-core/cubes/admin/cockpit/manifest.mjs
+7
View File
@@ -0,0 +1,7 @@
import { Manifest } from '@bitsquare/nopy-cube';
export default Manifest({
id: 'admin:cockpit',
name: 'Install cockpit and utils',
dependencies: () => [],
});
packages/cubes-core/cubes/admin/hostname/deploy.py
+34
View File
@@ -0,0 +1,34 @@
from pyinfra import host
from pyinfra.operations import server, files
"""
Deployment script for the admin:hostname cube.
Uses pyinfra's server.hostname operation to set and persist the system hostname.
"""
HOSTNAME = host.data.HOSTNAME
if HOSTNAME:
server.hostname(
name=f"Set system hostname to {HOSTNAME}",
hostname=HOSTNAME,
_sudo=True,
)
# 2. Update /etc/hosts to prevent "unable to resolve host" errors
# This looks for the line starting with 127.0.1.1 and replaces it entirely
files.line(
name="Update /etc/hosts for local resolution",
path="/etc/hosts",
line=r"^127\.0\.1\.1\s+.*",
replace=f"127.0.1.1 {HOSTNAME}",
_sudo=True,
)
# 3. Restart Avahi (Network Broadcast)
# This pushes the name change out to the shared network
server.service(
name="Restart Avahi to broadcast new mDNS name",
service="avahi-daemon",
restarted=True,
_sudo=True,
)
packages/cubes-core/cubes/admin/hostname/manifest.mjs
+20
View File
@@ -0,0 +1,20 @@
import { Manifest, uniqid } from '@bitsquare/nopy-cube';
import { z } from 'zod';
/**
* Manifest for the admin:hostname cube.
* This cube allows for setting and persistently changing the system's hostname.
*/
export default Manifest({
id: 'admin:hostname',
name: 'Permanently change the hostname',
dependencies: () => [],
schema: z.object({
HOSTNAME: z
.string()
.min(1)
.max(64)
.describe('The new hostname for the target host')
.default(`host-${uniqid()}`),
}),
});
packages/cubes-core/cubes/admin/locale/README.md
+23
View File
@@ -0,0 +1,23 @@
# Cube: admin/locale
Configures system keyboard layout permanently by updating `/etc/default/keyboard` and using `localectl`.
## Configuration
- `LAYOUT` (string): Keyboard layout (e.g. "ch", "us", "de"). Default: "ch".
- `MODEL` (string): Keyboard model. Default: "pc105".
- `VARIANT` (string): Keyboard variant. Default: "".
- `OPTIONS` (string): Keyboard options (comma separated). Default: "".
## Usage
```javascript
import { Manifest } from '@bitsquare/nopy-cube';
export default Manifest({
name: 'My Host Setup',
dependencies: () => [
['admin:locale', { LAYOUT: 'de' }]
]
});
```
packages/cubes-core/cubes/admin/locale/deploy.py
+55
View File
@@ -0,0 +1,55 @@
from pyinfra import host
from pyinfra.operations import server, files
LAYOUT = host.data.LAYOUT
MODEL = host.data.MODEL
VARIANT = host.data.VARIANT
OPTIONS = host.data.OPTIONS
# Update /etc/default/keyboard
files.line(
name="Update XKBMODEL in /etc/default/keyboard",
path="/etc/default/keyboard",
line=r'^XKBMODEL=.*',
replace=f'XKBMODEL="{MODEL}"',
_sudo=True,
)
files.line(
name="Update XKBLAYOUT in /etc/default/keyboard",
path="/etc/default/keyboard",
line=r'^XKBLAYOUT=.*',
replace=f'XKBLAYOUT="{LAYOUT}"',
_sudo=True,
)
files.line(
name="Update XKBVARIANT in /etc/default/keyboard",
path="/etc/default/keyboard",
line=r'^XKBVARIANT=.*',
replace=f'XKBVARIANT="{VARIANT}"',
_sudo=True,
)
files.line(
name="Update XKBOPTIONS in /etc/default/keyboard",
path="/etc/default/keyboard",
line=r'^XKBOPTIONS=.*',
replace=f'XKBOPTIONS="{OPTIONS}"',
_sudo=True,
)
# Apply keyboard configuration
server.shell(
name="Apply keyboard setup",
commands=["setupcon", "service keyboard-setup restart"],
_sudo=True,
)
# Set X11 keyboard layout using localectl if available
server.shell(
name="Set X11 keyboard layout using localectl",
commands=[f"localectl set-x11-keymap {LAYOUT} {MODEL} '{VARIANT}' '{OPTIONS}'"],
_sudo=True,
_ignore_errors=True,
)
packages/cubes-core/cubes/admin/locale/manifest.mjs
+14
View File
@@ -0,0 +1,14 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'admin:locale',
name: 'Configure system locale and keyboard layout',
dependencies: () => [],
schema: z.object({
LAYOUT: z.string().describe('Keyboard layout (e.g. "ch", "us", "de")').default('ch'),
MODEL: z.string().describe('Keyboard model').default('pc105'),
VARIANT: z.string().describe('Keyboard variant').default(''),
OPTIONS: z.string().describe('Keyboard options (comma separated)').default(''),
}),
});
packages/cubes-core/cubes/apt/essentials/README.md
+39
View File
@@ -0,0 +1,39 @@
# apt:essentials
**Install essential packages**
## Purpose
This cube installs a curated collection of essential development tools and utilities that are commonly needed for server environments and development workflows.
## What This Cube Does
Installs the following packages via apt:
- **fish** - User-friendly command-line shell with autosuggestions and syntax highlighting
- **ranger** - Terminal-based file manager with vi key bindings
- **golang-go** - Go programming language compiler and tools
- **build-essential** - Essential compilation tools (gcc, g++, make, etc.)
- **python3** - Python 3 interpreter and standard library
- **pkg-config** - Helper tool for compiling applications and libraries
- **age** - Modern file encryption tool with small explicit keys
## Configuration
### Parameters
- **UPDATE** (boolean, default: `true`)
- If already installed, should packages be updated?
- Set to `false` to skip package cache updates and only install missing packages
## Dependencies
None - this cube can run standalone.
## Use Cases
This cube is ideal as a base dependency for other cubes that require:
- Basic development tools
- Modern shell environments
- File encryption capabilities
- Python or Go runtimes
packages/cubes-core/cubes/apt/essentials/deploy.py
+18
View File
@@ -0,0 +1,18 @@
from pyinfra.operations import apt
from pyinfra import host
UPDATE = host.data.UPDATE
apt.packages(
name='Install essentials',
packages=[
'fish',
'build-essential',
'python3',
'pkg-config',
'age'
],
update=UPDATE,
_sudo=True
)
packages/cubes-core/cubes/apt/essentials/manifest.mjs
+11
View File
@@ -0,0 +1,11 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'apt:essentials',
name: 'Install essential packages',
dependencies: () => [],
schema: z.object({
UPDATE: z.boolean().describe('If already installed should packages be updated').default(true),
}),
});
packages/cubes-core/cubes/apt/install/README.md
+62
View File
@@ -0,0 +1,62 @@
# apt
**Install packages with apt**
## Purpose
This is a generic cube for installing custom packages via the apt package manager. It's useful when you need to install specific packages that aren't covered by other specialized cubes.
## What This Cube Does
1. Optionally updates the apt package cache
2. Installs the specified space-separated list of packages
## Configuration
### Parameters
- **UPDATE** (boolean, default: `true`)
- Update package cache before installing
- Set to `false` to skip updating and only install packages
- **PACKAGES** (string, default: `''`)
- Space-separated list of packages to install
- Example: `"vim git htop curl wget"`
## Dependencies
None - this cube can run standalone.
## Use Cases
Install development tools:
```
PACKAGES="vim git htop tmux"
```
Install database clients:
```
PACKAGES="postgresql-client mysql-client redis-tools"
```
Install system utilities:
```
PACKAGES="curl wget jq unzip zip"
```
## Example
When deploying this cube, you would typically configure it like:
```javascript
exec('apt', {
PACKAGES: 'nginx certbot python3-certbot-nginx',
UPDATE: true
})
```
## Notes
- Package names must match exact apt package names
- Invalid package names will cause the installation to fail
- Use `apt search <package>` to find package names
- Some packages may require additional configuration after installation
packages/cubes-core/cubes/apt/install/deploy.py
+31
View File
@@ -0,0 +1,31 @@
from pyinfra.operations import apt
from pyinfra import host
UPDATE = host.data.UPDATE
PACKAGES = str(host.data.PACKAGES).split(' ')
apt.packages(
name='Install essential packages',
packages=[
'htop',
'age',
'git',
'curl',
'nano',
'wget',
'ca-certificates',
'ufw',
"gnupg",
"lsb-release"
],
update=UPDATE,
_sudo=True
)
apt.packages(
name='Install custom packages',
packages=[p.strip() for p in PACKAGES if p],
update=UPDATE,
_sudo=True
)
packages/cubes-core/cubes/apt/install/manifest.mjs
+15
View File
@@ -0,0 +1,15 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'apt:install',
name: 'Install packages with apt',
dependencies: () => [],
schema: z.object({
UPDATE: z.boolean().describe('Update package cache before installing').default(false),
PACKAGES: z
.string()
.describe('Space-separated list of packages to install')
.default('vim htop curl'),
}),
});
packages/cubes-core/cubes/armor/fail2ban/README.md
+41
View File
@@ -0,0 +1,41 @@
# armor-fail2ban
**Install and enable fail2ban**
## Purpose
This cube installs and configures Fail2ban, an intrusion prevention software that protects your server from brute-force attacks and unauthorized access attempts.
## What is Fail2ban?
Fail2ban monitors log files (e.g., `/var/log/auth.log`) for suspicious activity, such as repeated failed login attempts. When it detects malicious behavior patterns, it automatically:
- Bans the offending IP address by updating firewall rules
- Prevents the attacker from making further connection attempts
- Can send email notifications about bans (if configured)
Common use cases include:
- Protecting SSH from brute-force password attacks
- Blocking repeated failed login attempts on web applications
- Preventing DoS attacks from specific IP addresses
## What This Cube Does
1. Installs the `fail2ban` package via apt
2. Deploys a custom configuration file (`jail.local`) to `/etc/fail2ban/jail.local`
3. Configures fail2ban with sensible defaults for common services
## Configuration
This cube currently has no configurable parameters. The default configuration is applied from the included `jail.local` file.
## Dependencies
None - this cube can run standalone.
## Notes
After deployment, you can:
- Check fail2ban status: `sudo fail2ban-client status`
- View banned IPs: `sudo fail2ban-client status sshd`
- Unban an IP: `sudo fail2ban-client set sshd unbanip <IP_ADDRESS>`
packages/cubes-core/cubes/armor/fail2ban/deploy.py
+18
View File
@@ -0,0 +1,18 @@
from pyinfra.operations import apt, files, server
# Install Fail2ban
apt.packages(
name='Install Fail2ban',
packages=['fail2ban'],
update=True,
_sudo=True
)
# Configure Fail2ban
files.put(
name='Configure Fail2ban',
src='jail.local',
dest='/etc/fail2ban/jail.local',
mode='0644',
_sudo=True
)
packages/cubes-core/cubes/armor/fail2ban/jail.local
+8
View File
@@ -0,0 +1,8 @@
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 600
bantime = 3600
packages/cubes-core/cubes/armor/fail2ban/manifest.mjs
+7
View File
@@ -0,0 +1,7 @@
import { Manifest } from '@bitsquare/nopy-cube';
export default Manifest({
id: 'armor:fail2ban',
name: 'Install and enable fail2ban',
dependencies: () => [],
});
packages/cubes-core/cubes/armor/ssh/README.md
+83
View File
@@ -0,0 +1,83 @@
# armor-ssh
**Secure SSH server by disabling password authentication**
## Purpose
This cube hardens your SSH server configuration by disabling less secure authentication methods, enforcing SSH key-based authentication only.
## Why Disable Password Authentication?
Password-based SSH authentication is vulnerable to:
- **Brute-force attacks**: Automated scripts trying millions of password combinations
- **Dictionary attacks**: Guessing common passwords
- **Credential stuffing**: Using leaked passwords from other breaches
- **Weak passwords**: Users choosing easily guessable passwords
**SSH key authentication is more secure** because:
- Keys are cryptographically strong (2048+ bit keys vs 8-12 character passwords)
- Private keys never travel over the network
- Immune to brute-force attacks
- Can be protected with passphrases for additional security
## What This Cube Does
1. **Disables challenge-response authentication**
- Prevents keyboard-interactive authentication prompts
2. **Optionally disables password authentication** (default: enabled)
- Forces users to authenticate with SSH keys only
- Prevents password-based login attempts
3. **Optionally disables PAM** (Pluggable Authentication Modules)
- Disables PAM-based authentication methods
- Reduces attack surface
4. **Restarts SSH service**
- Applies the new configuration immediately
## Configuration
### Parameters
- **DISABLE_PASSWORD** (boolean, default: `true`)
- Disable password authentication for SSH connections
- ⚠️ **WARNING**: Ensure you have SSH key access configured before enabling this!
- **DISABLE_PAM** (boolean, default: `true`)
- Disable PAM (Pluggable Authentication Modules) for SSH
- Recommended for key-only authentication setups
## Dependencies
None - this cube can run standalone.
## Security Best Practices
**Before deploying this cube**:
1. Ensure you have SSH key authentication set up and tested
2. Keep an alternative access method available (console access, VNC, etc.)
3. Test SSH key login before disabling passwords
4. Consider using the `user-add` or `ssh-keyman` cubes first
**After deployment**:
- Only SSH key authentication will work
- Password login attempts will be rejected
- Make sure to back up your private SSH key securely
## Post-Installation
The SSH service will restart automatically. Your current SSH session will remain active, but new connections must use SSH keys.
To verify the configuration:
```bash
sudo grep -E "PasswordAuthentication|ChallengeResponseAuthentication|UsePAM" /etc/ssh/sshd_config
```
## Recovery
If you get locked out:
1. Access the server via console (physical or cloud provider's web console)
2. Edit `/etc/ssh/sshd_config`
3. Set `PasswordAuthentication yes`
4. Restart SSH: `sudo systemctl restart ssh`
packages/cubes-core/cubes/armor/ssh/deploy.py
+43
View File
@@ -0,0 +1,43 @@
from pyinfra.operations import files, server
from pyinfra import host
from pyinfra import config
import logging
DISABLE_PASSWORD=host.data.DISABLE_PASSWORD
DISABLE_PAM=host.data.DISABLE_PAM
logger = logging.getLogger(__name__)
config.SUDO = True
files.line(
name='Disable challenge-response authentication in SSH',
path='/etc/ssh/sshd_config',
line='ChallengeResponseAuthentication yes',
replace='ChallengeResponseAuthentication no',
)
if DISABLE_PASSWORD:
files.line(
name='Disable password authentication in SSH',
path='/etc/ssh/sshd_config',
line='PasswordAuthentication yes',
replace='PasswordAuthentication no',
)
else:
logger.info('Password authentication allowed')
if DISABLE_PAM:
files.line(
name='Disable PAM in SSH',
path='/etc/ssh/sshd_config',
line='UsePAM yes',
replace='UsePAM no',
)
# Restart SSH service
server.service(
'ssh',
running=True,
restarted=True,
reloaded=True
)
packages/cubes-core/cubes/armor/ssh/manifest.mjs
+18
View File
@@ -0,0 +1,18 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'armor:ssh',
name: 'Secure SSH server by disabling password authentication',
dependencies: () => [],
schema: z.object({
DISABLE_PASSWORD: z
.boolean()
.describe('Disable password authentication for SSH connections')
.default(true),
DISABLE_PAM: z
.boolean()
.describe('Disable PAM (Pluggable Authentication Modules) for SSH')
.default(true),
}),
});
packages/cubes-core/cubes/armor/ufw/README.md
+77
View File
@@ -0,0 +1,77 @@
# armor-ufw
**Activate UFW (Uncomplicated Firewall)**
## Purpose
This cube configures and enables UFW, a user-friendly firewall management tool for Linux systems, providing basic protection against unauthorized network access.
## What is UFW?
UFW (Uncomplicated Firewall) is a frontend for `iptables` designed to make firewall configuration simple and accessible. It provides:
- **Easy-to-understand syntax**: Commands like `ufw allow ssh` instead of complex iptables rules
- **Default deny policy**: Blocks all incoming connections except those explicitly allowed
- **Connection tracking**: Automatically handles related and established connections
- **Application profiles**: Pre-configured rules for common services
Think of UFW as a security gate for your server - it controls which network traffic is allowed in and out.
## What This Cube Does
1. Configures UFW to allow SSH connections (port 22)
- Ensures you don't lock yourself out when enabling the firewall
2. Optionally allows HTTP traffic (port 80) based on the `ALLOW_HTTP` parameter
3. Enables the firewall with the configured rules
## Configuration
### Parameters
- **ALLOW_HTTP** (boolean, default: `true`)
- Allow incoming HTTP traffic on port 80
- Set to `false` if you're only using HTTPS or don't need web traffic
## Dependencies
- **apt:essentials** - Required for basic system tools
## Security Notes
**Important**: This cube automatically allows SSH to prevent lockouts. If you need to allow additional services, you can run:
```bash
sudo ufw allow [port number]/[protocol]
sudo ufw allow [service-name]
```
Examples:
- `sudo ufw allow 443/tcp` - Allow HTTPS
- `sudo ufw allow 3000/tcp` - Allow custom application port
- `sudo ufw allow https` - Allow HTTPS by service name
## Post-Installation
Check firewall status:
```bash
sudo ufw status verbose
sudo ufw status numbered
```
Common UFW commands:
- Delete rule: `sudo ufw delete [rule number]`
- Disable firewall: `sudo ufw disable`
- Reset to defaults: `sudo ufw reset`
## UFW File Locations
UFW rules are stored in the `/etc/ufw` directory:
- `/etc/ufw/user.rules` - Custom rules added via the `ufw` command
- `/etc/ufw/before.rules` - Rules processed before user rules (high priority)
- `/etc/ufw/after.rules` - Rules processed after user rules (exceptions)
- `/etc/ufw/sysctl.conf` - Kernel network parameters (e.g., packet forwarding)
- `/etc/ufw/applications.d/` - Application profiles for common services
- `/etc/default/ufw` - Global UFW settings and default policies
Understanding these locations is helpful for troubleshooting, manual edits, or backing up your firewall configuration.
packages/cubes-core/cubes/armor/ufw/deploy.py
+13
View File
@@ -0,0 +1,13 @@
from pyinfra.operations import server
from pyinfra import host
ALLOW_HTTP=host.data.ALLOW_HTTP
server.shell(
commands=[
f"ufw allow ssh",
f"ufw allow http" if ALLOW_HTTP else "",
f"ufw enable",
],
_sudo=True
)
packages/cubes-core/cubes/armor/ufw/manifest.mjs
+11
View File
@@ -0,0 +1,11 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'armor:ufw',
name: 'Activate ufw (uncomplicated firewall)',
dependencies: () => ['apt:essentials'],
schema: z.object({
ALLOW_HTTP: z.boolean().describe('Allow incoming HTTP traffic on port 80').default(true),
}),
});
packages/cubes-core/cubes/caddy/base/README.md
+120
View File
@@ -0,0 +1,120 @@
# caddy
**Install Caddy webserver**
## Purpose
This cube installs Caddy, a modern, powerful web server with automatic HTTPS that's designed to be easy to use and configure.
## What is Caddy?
Caddy is a next-generation web server that stands out for its simplicity and built-in security:
- **Automatic HTTPS**: Automatically obtains and renews SSL/TLS certificates from Let's Encrypt
- **Modern HTTP features**: HTTP/2, HTTP/3 (QUIC) support out of the box
- **Simple configuration**: Human-readable Caddyfile format
- **Reverse proxy**: Easy proxying to backend applications
- **Static file serving**: Fast and efficient static site hosting
- **Zero-downtime reloads**: Update config without dropping connections
## What This Cube Does
1. **Adds Caddy's official repository**
- Installs required dependencies (debian-keyring, apt-transport-https, curl)
- Downloads and installs Caddy's GPG signing key
- Configures APT to use Caddy's official stable repository
2. **Installs Caddy**
- Installs the latest stable version of Caddy
- Sets up the Caddy service
3. **Configures TLS**
- Creates a Caddyfile with a reusable TLS snippet
- Configures TLS based on the `TLS` parameter
## Configuration
### Parameters
- **TLS** (string, default: `''`)
- TLS certificate configuration
- **Options**:
- `''` (empty string) - Automatic HTTPS with Let's Encrypt (recommended)
- `'internal'` - Use Caddy's internal CA for self-signed certs (testing only)
- `'/path/to/cert /path/to/key'` - Provide custom certificate paths
## Dependencies
None - this cube can run standalone.
## TLS Configuration Examples
**Automatic HTTPS (Production)**:
```javascript
exec('caddy', { TLS: '' })
```
Caddy will automatically obtain SSL certificates from Let's Encrypt for your domain.
**Self-Signed for Testing**:
```javascript
exec('caddy', { TLS: 'internal' })
```
Uses Caddy's internal CA. Browsers will show security warnings.
**Custom Certificates**:
```javascript
exec('caddy', { TLS: '/etc/ssl/certs/mycert.pem /etc/ssl/private/mykey.pem' })
```
Use your own certificate and private key files.
## Post-Installation
The Caddyfile is created at `/etc/caddy/Caddyfile` with a reusable TLS snippet:
```
(tls_cert) {
tls {TLS}
}
```
Other cubes (like `caddy-spa`) can import this snippet with `import tls_cert`.
## Managing Caddy
Start/stop/restart Caddy:
```bash
sudo systemctl start caddy
sudo systemctl stop caddy
sudo systemctl restart caddy
sudo systemctl status caddy
```
Reload configuration without downtime:
```bash
sudo systemctl reload caddy
```
Test configuration:
```bash
caddy validate --config /etc/caddy/Caddyfile
```
## Common Use Cases
- Reverse proxy for Node.js/Python/Go apps
- Static website hosting
- API gateway
- Load balancer
- SSL/TLS termination
## Notes
- Caddy runs on ports 80 (HTTP) and 443 (HTTPS) by default
- Ensure these ports are open in your firewall (UFW)
- For automatic HTTPS, your domain must point to your server's IP
- Caddy automatically redirects HTTP to HTTPS when using automatic HTTPS
## Additional Resources
- [Caddy Documentation](https://caddyserver.com/docs/)
- [Caddyfile Tutorial](https://caddyserver.com/docs/caddyfile-tutorial)
packages/cubes-core/cubes/caddy/base/deploy.py
+34
View File
@@ -0,0 +1,34 @@
from pyinfra.operations import apt, server, files
from pyinfra import host
from io import StringIO
# 🔹 Variables
TLS = host.data.TLS
server.shell(
commands=[
f"sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl",
f"curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg",
f"curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list"
]
)
apt.packages(
name='Install caddy',
packages=['caddy'],
update=True,
_sudo=True
)
TLS_BLOCK = f"""
(tls_cert) {{
tls {TLS}
}}
"""
files.put(
src = StringIO(TLS_BLOCK), # local filename to upload,
dest = '/etc/caddy/Caddyfile', # the remote filename to upload to
_sudo=True
)
packages/cubes-core/cubes/caddy/base/manifest.mjs
+19
View File
@@ -0,0 +1,19 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'caddy',
name: 'Install Caddy webserver',
dependencies: () => [],
schema: z.object({
// tls /path/to/cert.pem /path/to/key.pem
TLS: z
.string()
.default('')
.describe(`
<empty-string> --> enabled and automatically managed
internal --> use Caddy custom root CA for self-signed certs (for testing purpose)
/path/to/cert /path/to/key --> Provide custom certificates
`),
}),
});
packages/cubes-core/cubes/caddy/spa/README.md
+137
View File
@@ -0,0 +1,137 @@
# caddy-spa
**Configure Caddy to serve a Single Page Application**
## Purpose
This cube adds a reverse proxy configuration to Caddy for serving a Single Page Application (SPA) from a local backend server, with automatic HTTPS support.
## What This Cube Does
1. **Adds domain configuration to Caddyfile**
- Creates a site block for the specified domain
- Configures reverse proxy to forward traffic to local application
- Imports TLS configuration from the `caddy` cube
2. **Configures reverse proxy**
- Proxies all requests to `localhost:{PORT}`
- Preserves headers and client information
- Handles WebSocket connections
3. **Restarts Caddy service**
- Applies the new configuration immediately
## Configuration
### Parameters
- **DOMAIN** (string, default: `''`)
- Domain name for the SPA application
- Example: `'myapp.example.com'`
- Must have DNS pointing to your server's IP
- **PORT** (number, default: `5432`)
- Port number where the SPA will be served
- Your application should be listening on this port locally
## Dependencies
- **caddy** cube (implicitly required) - Must be installed first to provide the `tls_cert` snippet
## What Gets Configured
This cube adds the following to `/etc/caddy/Caddyfile`:
```
# BEGIN DOMAIN myapp.example.com
myapp.example.com {
import tls_cert
reverse_proxy localhost:5432
}
# END myapp.example.com
```
## Use Cases
**Deploy a React/Vue/Angular app**:
```javascript
exec('caddy-spa', {
DOMAIN: 'app.example.com',
PORT: 3000
})
```
**Deploy multiple SPAs**:
```javascript
exec('caddy-spa', { DOMAIN: 'app1.example.com', PORT: 3000 })
exec('caddy-spa', { DOMAIN: 'app2.example.com', PORT: 3001 })
exec('caddy-spa', { DOMAIN: 'app3.example.com', PORT: 3002 })
```
## How It Works
1. User visits `https://myapp.example.com`
2. Caddy receives the request on port 443 (HTTPS)
3. Caddy automatically handles SSL/TLS encryption
4. Request is forwarded to `localhost:5432`
5. Your application receives the request and returns a response
6. Caddy sends the encrypted response back to the user
## Prerequisites
Before deploying this cube:
1. **Install the caddy cube first**
- Provides the base Caddy installation and TLS configuration
2. **Ensure your application is running**
- Your SPA backend should be listening on the specified PORT
- Example: `npm start` or `pm2 start app.js`
3. **Configure DNS**
- Point your domain's A record to your server's IP address
- Wait for DNS propagation (can take a few minutes to hours)
4. **Open firewall ports**
- Ensure ports 80 and 443 are open (for automatic HTTPS)
- `sudo ufw allow 80/tcp`
- `sudo ufw allow 443/tcp`
## Post-Installation
Verify the configuration:
```bash
sudo caddy validate --config /etc/caddy/Caddyfile
```
Check Caddy status:
```bash
sudo systemctl status caddy
```
View Caddy logs:
```bash
sudo journalctl -u caddy -f
```
## Common Issues
**502 Bad Gateway**:
- Your application isn't running on the specified PORT
- Check: `netstat -tlnp | grep {PORT}`
**Certificate errors**:
- DNS not pointing to your server
- Ports 80/443 blocked by firewall
- Check Caddy logs: `sudo journalctl -u caddy -f`
**Domain not resolving**:
- DNS propagation not complete yet
- Verify with: `dig +short myapp.example.com`
## Notes
- Caddy automatically obtains and renews Let's Encrypt certificates
- The reverse proxy preserves the original client IP and headers
- WebSocket connections are automatically supported
- You can add multiple domains by running this cube multiple times with different parameters
packages/cubes-core/cubes/caddy/spa/deploy.py
+32
View File
@@ -0,0 +1,32 @@
from pyinfra.operations import apt, server, files
from pyinfra import host
from io import StringIO
# 🔹 Variables (Modify as Needed)
DOMAIN = host.data.DOMAIN
PORT = host.data.PORT
site_block = f"""# BEGIN DOMAIN {DOMAIN}
{DOMAIN} {{
import tls_cert
reverse_proxy localhost:{PORT}
}}
# END {DOMAIN}
"""
files.block(
path = '/etc/caddy/Caddyfile',
content = site_block,
present = True,
before = False,
after = False,
_sudo = True
)
server.service(
service='caddy',
running=True,
restarted=True,
_sudo=True
)
packages/cubes-core/cubes/caddy/spa/manifest.mjs
+12
View File
@@ -0,0 +1,12 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'caddy:spa',
name: 'Install single page application',
dependencies: () => [],
schema: z.object({
DOMAIN: z.string().describe('Domain name for the SPA application').default(''),
PORT: z.number().describe('Port number where the SPA will be served').default(5432),
}),
});
packages/cubes-core/cubes/git/clone/deploy.py
+18
View File
@@ -0,0 +1,18 @@
from pyinfra.operations import server
from pyinfra import host
USER = host.data.USER
REPO = host.data.REPO
APP = host.data.APP
server.shell(
name="Clone application",
commands=[
f"""
cd $HOME &&
git clone --recurse-submodules {REPO} {APP}
"""],
_sudo=True,
_sudo_user=USER,
_use_sudo_login=True
)
packages/cubes-core/cubes/git/clone/manifest.mjs
+13
View File
@@ -0,0 +1,13 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'git:clone',
name: 'Clone a repository',
dependencies: () => [],
schema: z.object({
USER: z.string().describe('Username for which to clone the repository').default('vagrant'),
REPO: z.string().default(''),
APP: z.string().describe('The internal name used for this application').default(''),
}),
});
packages/cubes-core/cubes/network/tailscale/README.md
+33
View File
@@ -0,0 +1,33 @@
# Tailscale Cube
Installs and authenticates the Tailscale client on a Linux host.
## Features
- **Automated Installation**: Adds the official Tailscale repository and installs the package.
- **Headless Authentication**: Uses a Tailscale Auth Key for zero-interaction setup.
- **Headscale Support**: Can be configured to connect to a custom login server.
- **Startup persistence**: Ensures the `tailscaled` daemon is enabled and running.
## Configuration
| Variable | Default | Description |
|----------|---------|-------------|
| `AUTH_KEY` | `""` | **Secret.** Tailscale Auth Key (recommended to use a 'reusable' or 'ephemeral' key). |
| `LOGIN_SERVER` | `https://controlplane.tailscale.com` | The coordination server URL. Set this to your Headscale instance URL if applicable. |
| `EXTRA_ARGS` | `""` | Additional flags to pass to `tailscale up` (e.g., `--advertise-exit-node`). |
| `FORCE_REAUTH` | `false` | If true, forces the client to re-authenticate. |
## Usage
```bash
nopy install tailscale
```
When prompted, provide your `AUTH_KEY`. If you are using Headscale, also provide the `LOGIN_SERVER` URL.
`AUTH_KEY` is declared in the manifest's `secrets`, so nopy keeps it out of session
and history files and masks it in any command it prints. It is asked for again on
replay, and a `--use-defaults` replay refuses rather than joining the tailnet with
an empty key. Prefer an ephemeral key regardless — the value is still on pyinfra's
command line while the deployment runs.
packages/cubes-core/cubes/network/tailscale/deploy.py
+45
View File
@@ -0,0 +1,45 @@
from pyinfra import host
from pyinfra.operations import server, apt
# Variables from manifest
AUTH_KEY = host.data.AUTH_KEY
LOGIN_SERVER = host.data.LOGIN_SERVER
EXTRA_ARGS = host.data.EXTRA_ARGS
FORCE_REAUTH = host.data.FORCE_REAUTH
# 1. Install Tailscale using the official one-liner script
server.shell(
name="Install Tailscale",
commands=["curl -fsSL https://tailscale.com/install.sh | sh"],
_sudo=True
)
# 2. Ensure Tailscale is enabled and running
# Manually (Linux): sudo systemctl enable --now tailscaled
server.service(
name="Ensure tailscaled is running and enabled on boot",
service="tailscaled",
running=True,
enabled=True,
_sudo=True
)
# 3. Authenticate and bring Tailscale up
# We use --authkey for headless mode
# We use --login-server if it's different from the default
up_command = f"tailscale up --authkey {AUTH_KEY}"
if LOGIN_SERVER and LOGIN_SERVER != "https://controlplane.tailscale.com":
up_command += f" --login-server {LOGIN_SERVER}"
if FORCE_REAUTH:
up_command += " --force-reauth"
if EXTRA_ARGS:
up_command += f" {EXTRA_ARGS}"
server.shell(
name="Authenticate Tailscale (Headless)",
commands=[up_command],
_sudo=True
)
packages/cubes-core/cubes/network/tailscale/manifest.mjs
+18
View File
@@ -0,0 +1,18 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'net:tailscale',
name: 'Install and authenticate Tailscale',
dependencies: () => ['apt:essentials'],
secrets: ['AUTH_KEY'],
schema: z.object({
AUTH_KEY: z.string().describe('Tailscale Auth Key for headless authentication').default(''),
LOGIN_SERVER: z
.string()
.describe('Custom login server (e.g., for Headscale)')
.default('https://controlplane.tailscale.com'),
EXTRA_ARGS: z.string().describe('Additional arguments for tailscale up').default(''),
FORCE_REAUTH: z.boolean().describe('Force re-authentication').default(false),
}),
});
packages/cubes-core/cubes/network/wifi/access-point/README.md
+243
View File
@@ -0,0 +1,243 @@
# WiFi Access Point Cube (NetworkManager)
Configures a Linux device as a WiFi Access Point using NetworkManager's `nmcli` command. **This approach allows the device to simultaneously act as an AP AND remain connected to another WiFi network as a client.**
## Key Features
- ✅ **Dual WiFi Mode**: Acts as AP while staying connected to another WiFi
- ✅ **Modern Approach**: Uses NetworkManager (nmcli) instead of hostapd
- ✅ **Auto-start**: Configured to start on boot
- ✅ **IP Sharing**: Built-in internet sharing with `ipv4.method shared`
- ✅ **Simple Configuration**: Minimal parameters, maximum functionality
- ✅ **Supports 2.4GHz and 5GHz bands**
## Requirements
- Device with WiFi capability (e.g., Raspberry Pi 4/5)
- NetworkManager installed and running
- WiFi hardware that supports AP mode
## Configuration Parameters
> **This section is out of date** — it lists parameters the manifest does not
> declare (`NETWORK_DEVICE`, `CHANNEL`, `IP_ADDRESS`) and omits `AP_IP`. Read
> `manifest.mjs` for the real list. Tracked as §5 in the repository's
> `DOCS-AUDIT.md`.
### Prompted first
- **SSID**: WiFi network name (1-32 characters). Defaults to `PiPoint`.
- **PASSWORD**: WPA2 password (8-63 characters). Defaults to `1223334444` — a
placeholder that should not survive contact with a real network.
Declared in the manifest's `secrets`, so nopy keeps it out of session and
history files and masks it in printed commands, and re-prompts on replay. The
value is still on pyinfra's command line, so it is visible in `ps` during the run.
### Optional (with defaults)
- **NETWORK_DEVICE**: WiFi interface to use (`wlan0` default)
- **BAND**: Frequency band - `2.4GHz` or `5GHz` (auto-detected from current connection if not specified)
- **CHANNEL**: WiFi channel (auto-detected from current connection if not specified)
- 2.4GHz: 1-14
- 5GHz: 36-165
- **IP_ADDRESS**: AP gateway address (`192.168.50.1` default)
- **CONNECTION_NAME**: NetworkManager connection name (`net:wifi:ap` default)
## Example Usage
### Basic Configuration
```json
{
"SSID": "MyHomeWiFi",
"PASSWORD": "SecurePass123"
}
```
### Advanced Configuration (5GHz with specific channel)
```json
{
"SSID": "FastWiFi5G",
"PASSWORD": "SuperSecure456",
"NETWORK_DEVICE": "wlan1",
"BAND": "5GHz",
"CHANNEL": 36,
"IP_ADDRESS": "10.0.0.1",
"CONNECTION_NAME": "my-hotspot"
}
```
### Auto-Detection Configuration
When `BAND` and `CHANNEL` are not specified, the script will automatically detect the current AP's band and channel using `iw dev <interface> link` and use those values for the new hotspot. This is useful when you want the hotspot to operate on the same band/channel as your existing connection to avoid interference.
```json
{
"SSID": "MyAutoAP",
"PASSWORD": "SecurePass123"
}
```
## What This Cube Does
1. **Installs NetworkManager** (if not present)
2. **Enables NetworkManager service** and ensures it's running
3. **Auto-detects current AP settings** using `iw dev <interface> link` to find the band and channel
4. **Creates WiFi hotspot** using `nmcli` with auto-detected or specified settings
5. **Configures IP sharing** with `ipv4.method shared` (automatic DHCP + NAT)
6. **Enables IP forwarding** for internet routing
7. **Sets autoconnect** so the AP starts on boot
## How It Works (Dual WiFi Mode)
### Simultaneous AP + Client Mode
**Hardware Support Required:**
- Your WiFi hardware must support **simultaneous AP+STA (Station) mode**
- Most modern WiFi chips support this (e.g., Raspberry Pi 4/5, Intel WiFi cards)
- Older hardware may not support it (e.g., Raspberry Pi 3B and earlier have limitations)
**How NetworkManager Handles It:**
1. **Single Interface (e.g., wlan0):**
- If hardware supports AP+STA: ✅ **Both client and AP run on same interface**
- If hardware doesn't support it: ⚠️ **Client connection may be dropped**
2. **Multiple Interfaces (e.g., wlan0 + wlan1):**
- NetworkManager will use one for client, another for AP
- Always works regardless of hardware capabilities
**Example Scenario (Raspberry Pi 4):**
```
[Internet] <--WiFi--> [RPi4 wlan0 (Client+AP)] <--WiFi--> [Devices connect to AP]
```
**How to Check Hardware Support:**
```bash
iw list | grep -A 10 "valid interface combinations"
```
Look for: `* #{ managed } <= 1, #{ AP } <= 1` or `* #{ managed, AP } <= 2`
NetworkManager will:
- Share internet from **any** available source (WiFi client, Ethernet, cellular, etc.)
- Automatically handle routing and NAT
- Try to maintain client connection if hardware supports it
## Post-Installation
After deployment, the device will:
- ✅ Broadcast the WiFi network with your SSID
- ✅ Accept connections with your password
- ✅ Assign IP addresses to connected clients (via built-in DHCP)
- ✅ Share internet connection from any available interface
- ✅ Auto-start the AP on every boot
- ✅ Maintain client WiFi connection (if connected to another network)
## Managing the Hotspot
### View connection status
```bash
nmcli connection show
```
### Stop the hotspot
```bash
sudo nmcli connection down net:wifi:ap
```
### Start the hotspot
```bash
sudo nmcli connection up net:wifi:ap
```
### Disable autostart
```bash
sudo nmcli connection modify net:wifi:ap connection.autoconnect no
```
### Delete the hotspot
```bash
sudo nmcli connection delete net:wifi:ap
```
## Troubleshooting
### AP doesn't start
- Check NetworkManager status: `sudo systemctl status NetworkManager`
- Verify WiFi interface exists: `nmcli device status`
- Check if AP mode is supported: `iw list | grep -A 10 "Supported interface modes"`
### Can't connect to AP
- Verify password is correct (8+ characters)
- Check channel compatibility with your devices
- Try switching between 2.4GHz and 5GHz bands
### No internet on clients
- Verify host device has internet: `ping 8.8.8.8`
- Check IP forwarding: `sysctl net.ipv4.ip_forward`
- NetworkManager should handle NAT automatically with `ipv4.method shared`
### Hotspot conflicts with client WiFi
- This shouldn't happen with NetworkManager
- If it does, check if hardware supports simultaneous AP+STA mode:
```bash
iw list | grep "valid interface combinations"
```
## Advantages Over Traditional Approach
| Feature | Traditional (hostapd) | NetworkManager (nmcli) |
|---------|----------------------|------------------------|
| **Dual WiFi** | ❌ No (conflicts with wpa_supplicant) | ✅ Yes (AP + client simultaneously) |
| **Configuration** | Complex (multiple files) | Simple (one command) |
| **DHCP** | Manual (dnsmasq) | Automatic |
| **NAT** | Manual (iptables) | Automatic |
| **Management** | Multiple services | Single service |
| **Dependencies** | hostapd, dnsmasq, iptables | NetworkManager only |
## Security Notes
- Always use a strong password (minimum 8 characters)
- WPA2 encryption is automatically enabled
- NetworkManager handles firewall rules automatically
## Compatibility
Tested on:
- Raspberry Pi 4/5 with built-in WiFi
- Ubuntu/Debian-based systems with NetworkManager
- Devices with WiFi hardware supporting AP mode
**Note**: Older Raspberry Pi models (3B and earlier) may have limitations with simultaneous AP + client mode due to hardware constraints.
## Troubleshooting
```bash
sudo nmcli con add type wifi con-name MyHotspot ifname wlan0 mode ap ssid YourNewAP ipv4.method shared wifi-sec.key-mgmt wpa-psk wifi-sec.psk "YourNewPassword"
sudo nmcli con up MyHotspot
sudo nmcli con down MyHotspot
sudo nmcli con mod MyHotspot wifi.band <band> wifi.channel <channel_number>
# Find the currently used band and channel
iw dev wlan0 link
```
packages/cubes-core/cubes/network/wifi/access-point/deploy.py
+70
View File
@@ -0,0 +1,70 @@
from pyinfra.operations import apt, server, systemd
from pyinfra import host
# Extract variables with proper defaults
SSID = host.data.get('SSID')
PASSWORD = host.data.get('PASSWORD')
BAND = host.data.get('BAND', '2.4GHz')
CHANNEL = host.data.get('CHANNEL') # Optional
AP_IP = host.data.get('AP_IP')
CONNECTION_NAME = host.data.get('CONNECTION_NAME')
# Determine band configuration
if BAND == '5GHz':
BAND = 'a'
default_channel = 36 if not CHANNEL else CHANNEL
else: # 2.4GHz
BAND = 'bg'
default_channel = 6 if not CHANNEL else CHANNEL
# Install NetworkManager if not present
apt.packages(
name='Install NetworkManager',
packages=['network-manager'],
update=True,
_sudo=True
)
# Ensure NetworkManager is running
systemd.service(
name='Ensure NetworkManager is running',
service='NetworkManager',
running=True,
enabled=True,
_sudo=True
)
# Check if connection already exists and delete it
server.shell(
name='Delete existing AP connection if present',
commands=[f'nmcli connection delete {CONNECTION_NAME} || true'],
_sudo=True
)
# Add Wifi AP connection
server.shell(
name='Detect WiFi interface',
commands=[
f'sudo nmcli con add type wifi con-name {CONNECTION_NAME} ifname wlan0 mode ap ssid {SSID} ipv4.method shared wifi-sec.key-mgmt wpa-psk wifi-sec.psk "{PASSWORD}"',
# CRITICAL: Force WPA2 (rsn) instead of WPA (wpa)
f'sudo nmcli connection modify {CONNECTION_NAME} wifi-sec.proto rsn',
f'sudo nmcli connection modify {CONNECTION_NAME} ipv4.addresses {AP_IP}/24',
# Use AES encryption (CCMP) instead of TKIP
f'sudo nmcli connection modify {CONNECTION_NAME} wifi-sec.pairwise ccmp',
f'sudo nmcli connection modify {CONNECTION_NAME} wifi-sec.group ccmp',
# Set band
f'sudo nmcli connection modify {CONNECTION_NAME} 802-11-wireless.band {BAND}',
# Enable autostart
f'sudo nmcli connection modify {CONNECTION_NAME} connection.autoconnect yes',
f'sudo nmcli connection modify {CONNECTION_NAME} connection.autoconnect-priority 10',
# Start the AP
f'sudo nmcli connection up {CONNECTION_NAME}'
],
_sudo=True
)
packages/cubes-core/cubes/network/wifi/access-point/manifest.mjs
+21
View File
@@ -0,0 +1,21 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'net:wifi:access-point',
name: 'Configure WiFi Access Point (NetworkManager)',
dependencies: () => [],
secrets: ['PASSWORD'],
schema: z.object({
SSID: z.string().min(1).max(32).default('PiPoint').describe('WiFi network name (SSID)'),
PASSWORD: z
.string()
.min(8)
.max(63)
.default('1223334444')
.describe('WPA2 password (8-63 characters)'),
BAND: z.enum(['2.4GHz', '5GHz']).default('2.4GHz').describe('Frequency band'),
AP_IP: z.string().default('192.168.4.1').describe('AP IP address'),
CONNECTION_NAME: z.string().default('pi-point').describe('NetworkManager connection name'),
}),
});
packages/cubes-core/cubes/network/wifi/connection/README.md
+59
View File
@@ -0,0 +1,59 @@
# network:wifi:connection
**Configure a WiFi client connection using NetworkManager**
## Purpose
This cube allows you to connect your target host (e.g., a Raspberry Pi or a laptop) to an existing WiFi network using `nmcli`.
## What This Cube Does
1. Ensures `network-manager` is installed and running
2. Removes any existing connection with the same name to avoid conflicts
3. Connects to the specified `SSID` using the provided `PASSWORD`
4. Configures the connection to automatically connect on boot (optional)
## Configuration
| Variable | Type | Description | Required | Default |
| :--- | :--- | :--- | :--- | :--- |
| `SSID` | `string` | The WiFi network name | Yes | - |
| `PASSWORD` | `string` | The WiFi password | Yes | - |
| `AUTOCONNECT` | `boolean` | Automatically connect to this network | No | `true` |
| `CONNECTION_NAME` | `string` | Name for the connection in NetworkManager | No | `SSID` |
## Dependencies
- `apt/essentials`: Basic system utilities.
## Usage
### Simple Connection
```bash
nopy install network:wifi:connection --env SSID="MyHomeWiFi" --env PASSWORD="mysecurepassword"
```
### Connection with Custom Name and No Autoconnect
```bash
nopy install network:wifi:connection --env SSID="OfficeWiFi" --env PASSWORD="password123" --env CONNECTION_NAME="Work" --env AUTOCONNECT=false
```
## Security Notes
- `PASSWORD` is declared in the manifest's `secrets`: nopy keeps it out of session
and history files and masks it in every command it prints. It is prompted for
again on replay.
- That covers what nopy writes, not everything. The value is still on pyinfra's
command line, so it is visible in `ps` while the deployment runs.
- WiFi passwords will be stored in `/etc/NetworkManager/system-connections/` on the target host.
- Passing passwords via `--env` may leave them in your local shell history.
## Troubleshooting
You can check the status of your WiFi connections on the target host with:
```bash
nmcli connection show
nmcli device status
```
packages/cubes-core/cubes/network/wifi/connection/deploy.py
+43
View File
@@ -0,0 +1,43 @@
from pyinfra import host
from pyinfra.operations import apt, server, systemd
# [agnt://cogen/cogen/network-wifi-connection-2]{cartridge: "ansiblings/cubes", action: "generated", status: "generated"}
"""
Deployment script for network:wifi:connection.
Uses nmcli to configure a WiFi client connection.
"""
SSID = host.data.SSID
PASSWORD = host.data.PASSWORD
AUTOCONNECT = "yes" if host.data.get('AUTOCONNECT', True) else "no"
CONNECTION_NAME = host.data.get('CONNECTION_NAME', SSID)
# Install NetworkManager if not present
apt.packages(
name='Install NetworkManager',
packages=['network-manager'],
update=True,
_sudo=True
)
# Ensure NetworkManager is running
systemd.service(
name='Ensure NetworkManager is running',
service='NetworkManager',
running=True,
enabled=True,
_sudo=True
)
# Add/Update the WiFi connection
# We delete first to ensure a clean state with the new password/settings
server.shell(
name=f"Configure WiFi connection for {SSID}",
commands=[
f'nmcli connection delete "{CONNECTION_NAME}" || true',
f'nmcli device wifi connect "{SSID}" password "{PASSWORD}" name "{CONNECTION_NAME}"',
f'nmcli connection modify "{CONNECTION_NAME}" connection.autoconnect {AUTOCONNECT}',
],
_sudo=True
)
packages/cubes-core/cubes/network/wifi/connection/manifest.mjs
+26
View File
@@ -0,0 +1,26 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
// [agnt://cogen/cogen/network-wifi-connection-1]{cartridge: "ansiblings/cubes", action: "generated", status: "generated"}
/**
* Manifest for the network:wifi:connection cube.
* Configures a WiFi client connection using NetworkManager (nmcli).
*/
export default Manifest({
id: 'net:wifi:connection',
name: 'network:wifi:connection - Connect to a WiFi network',
secrets: ['PASSWORD'],
schema: z.object({
SSID: z.string().min(1).describe('The SSID of the WiFi network to connect to'),
PASSWORD: z.string().min(8).describe('The password for the WiFi network'),
AUTOCONNECT: z
.boolean()
.default(true)
.describe('Whether to automatically connect to this network'),
CONNECTION_NAME: z
.string()
.optional()
.describe('Optional name for the connection (defaults to SSID)'),
}),
});
packages/cubes-core/cubes/runtime/docker/README.md
+60
View File
@@ -0,0 +1,60 @@
# docker
**Install Docker and tools**
## Purpose
This cube installs Docker Engine, Docker Compose, and popular Docker management tools to enable containerized application deployment and management.
## What is Docker?
Docker is a platform for developing, shipping, and running applications in containers. Containers package an application with all its dependencies, ensuring it runs consistently across different environments.
Key benefits:
- **Isolation**: Each container runs independently with its own filesystem, network, and processes
- **Portability**: Containers run the same way on any system that supports Docker
- **Efficiency**: Containers share the host OS kernel, making them lighter than virtual machines
- **Scalability**: Easily deploy and scale containerized applications
## What This Cube Does
1. **Adds Docker's official repository**
- Downloads and installs Docker's GPG key
- Configures APT to use Docker's official package repository
2. **Installs Docker components**
- `docker-ce` - Docker Community Edition engine
- `docker-ce-cli` - Command-line interface for Docker
- `containerd.io` - Container runtime
- `docker-buildx-plugin` - Extended build capabilities with BuildKit
- `docker-compose-plugin` - Tool for defining multi-container applications
3. **Installs management tools**
- **lazydocker** - Terminal UI for Docker and Docker Compose management
- **docker-ctop** - Container metrics and monitoring (top-like interface for containers)
## Configuration
This cube currently has no configurable parameters.
## Dependencies
None - this cube can run standalone.
## Post-Installation
After deployment:
- Add users to the `docker` group to run Docker without sudo: `sudo usermod -aG docker username`
- Start using Docker: `docker run hello-world`
- Use lazydocker for easy management: `lazydocker`
- Monitor containers: `ctop`
## Notes
The Docker daemon starts automatically on boot. You can manage it with systemd:
- Check status: `sudo systemctl status docker`
- Restart: `sudo systemctl restart docker`
## Additional Resources
- [Installation tutorial Ubuntu 24.04](https://www.cherryservers.com/blog/install-docker-ubuntu)
packages/cubes-core/cubes/runtime/docker/deploy.py
+75
View File
@@ -0,0 +1,75 @@
from pyinfra import host
from pyinfra.operations import server, apt
DISTRO = host.data.DISTRO
server.shell(
commands=[
"install -m 0755 -d /etc/apt/keyrings",
f"curl -fsSL https://download.docker.com/linux/{DISTRO}/gpg | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg",
"sudo chmod a+r /etc/apt/keyrings/docker.gpg",
f""" echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/{DISTRO} "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null """,
"apt update"
],
_sudo=True
)
apt.packages(
packages=[
"docker-ce",
"docker-ce-cli",
"containerd.io",
"docker-buildx-plugin",
"docker-compose-plugin",
],
present=True,
_sudo=True
)
server.shell(
name="Install Lazydocker system-wide",
commands=[
'export DIR=/usr/local/bin && curl -fsSL https://raw.githubusercontent.com/jesseduffield/lazydocker/master/scripts/install_update_linux.sh | bash -s --'
],
_sudo=True
)
# Install required packages
apt.packages(
name="Install required dependencies",
packages=["ca-certificates", "curl", "gnupg", "lsb-release"],
update=True,
_sudo=True
)
# Download and store the GPG key
server.shell(
name="Download and store Azlux repo GPG key",
commands=[
"curl -fsSL https://azlux.fr/repo.gpg.key | gpg --dearmor --yes -o /usr/share/keyrings/azlux-archive-keyring.gpg"
],
_sudo=True
)
# Add the repository to sources.list
server.shell(
name="Add Azlux repository to APT sources",
commands=[
""" echo "deb [arch="$(dpkg --print-architecture)" signed-by=/usr/share/keyrings/azlux-archive-keyring.gpg] http://packages.azlux.fr/debian bookworm main" | sudo tee /etc/apt/sources.list.d/azlux.list > /dev/null """,
],
_sudo=True
)
# Update APT and install docker-ctop
apt.update(
name="Update package lists",
_sudo=True
)
apt.packages(
name="Install docker-ctop",
packages=["docker-ctop"],
_sudo=True
)
packages/cubes-core/cubes/runtime/docker/manifest.mjs
+11
View File
@@ -0,0 +1,11 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'runtime:docker',
name: 'Install docker and tools',
dependencies: () => [],
schema: z.object({
DISTRO: z.enum(['ubuntu', 'debian']).default('ubuntu').describe('Linux distribution to target'),
}),
});
packages/cubes-core/cubes/runtime/nodevm/README.md
+83
View File
@@ -0,0 +1,83 @@
# nodevm
**Install Node.js with essential global packages**
## Purpose
This cube installs the latest LTS (Long Term Support) version of Node.js along with essential global npm packages commonly needed for development and deployment.
## What is Node.js?
Node.js is a JavaScript runtime built on Chrome's V8 engine that allows you to run JavaScript on the server. It's widely used for:
- Building web servers and APIs
- Command-line tools
- Build tools and task runners
- Real-time applications (chat, notifications)
- Microservices
## What This Cube Does
1. **Installs Node.js LTS**
- Downloads and runs the official NodeSource setup script
- Installs the latest LTS version of Node.js
- Includes npm (Node Package Manager)
2. **Installs build dependencies**
- `libssl-dev` - SSL/TLS libraries
- `libtool` - Library building tools
- `cmake` - Cross-platform build system
- `libpng-dev`, `libjpeg-dev`, `libvips-dev` - Image processing libraries
3. **Installs global npm packages**
- **npm@11.1.0** - Latest npm version
- **pm2** - Production process manager for Node.js apps
- **yarn** - Alternative package manager
- **local-web-server** - Local development web server
- **node-gyp** - Node.js native addon build tool
- **inquirer** - Interactive command-line prompts
- **execa** - Better child process execution
- **@dotenvx/dotenvx** - Environment variable management
## Configuration
This cube currently has no configurable parameters.
## Dependencies
None - this cube can run standalone.
## Post-Installation
Verify installation:
```bash
node --version
npm --version
```
Common commands:
- Run a Node.js app: `node app.js`
- Start with PM2: `pm2 start app.js`
- Install packages: `npm install <package>`
- Use yarn: `yarn add <package>`
## PM2 - Process Manager
PM2 is included for production deployments. Common PM2 commands:
```bash
pm2 start app.js # Start application
pm2 list # List running apps
pm2 stop app # Stop application
pm2 restart app # Restart application
pm2 logs # View logs
pm2 startup # Enable PM2 on boot
pm2 save # Save current process list
```
## Notes
- Node.js is installed system-wide
- Global packages are accessible to all users
- npm cache is stored in `~/.npm`
- Use `nvm` if you need multiple Node.js versions
packages/cubes-core/cubes/runtime/nodevm/deploy.py
+58
View File
@@ -0,0 +1,58 @@
from pyinfra.operations import server, apt, npm, python
from pyinfra import host
from pyinfra.facts.files import Directory
from pyinfra.facts.server import Which
hasNode = host.get_fact(Which, 'node')
VERSION = host.data.VERSION
ALIAS = host.data.ALIAS
GLOBAL_PACKAGES = host.data.GLOBAL_PACKAGES
USER = host.data.USER
apt.packages(
name=f'Install nodejs tools',
no_recommends=True,
packages=[
'build-essential',
'libssl-dev',
'libtool',
'cmake',
'libcairo2-dev',
'libpango1.0-de',
'libpng-dev',
'libgif-dev',
'libjpeg-dev',
'libvips-dev',
'librsvg2-dev',
'libpixman-1-dev',
],
_sudo = True,
)
server.shell(
commands=[
"curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash",
"omf install nvm",
f"nvm install {VERSION}",
f"nvm alias {ALIAS} {VERSION}",
"set -gx NVM_DIR $HOME/.nvm",
],
_sudo=True,
_su_user=USER,
_use_su_login=True,
_shell_executable='/usr/bin/fish'
)
server.shell(
commands=[
"npm install -g pm2 yarn local-web-server node-gyp inquirer execa @dotenvx/dotenvx"
],
_sudo=True,
_su_user=USER,
_use_su_login=True,
_shell_executable='/usr/bin/fish'
)
packages/cubes-core/cubes/runtime/nodevm/manifest.mjs
+20
View File
@@ -0,0 +1,20 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'runtime:nodevm',
name: 'Install nvm and nodejs with global packages',
dependencies: () => [],
schema: z.object({
VERSION: z
.nullable(z.string())
.describe('Node.js version to install. It is recommended to use semver notation')
.default('v22.20.0'),
USER: z.string().describe('Username for which to install nodejs').default('vagrant'),
ALIAS: z.string().describe('The alias for this node version').default('nodelts'),
GLOBAL_PACKAGES: z
.string()
.describe('Space-separated list of global npm packages to install')
.default('npm-check-updates'),
}),
});
packages/cubes-core/cubes/service/autostart/README.md
+176
View File
@@ -0,0 +1,176 @@
# TypeStack Install Cube
Deploys a Node.js/TypeScript application from a Git repository as a systemd service with Docker Compose and PM2 support.
## Features
- Clones Git repository
- Installs dependencies with Yarn
- Builds the application
- Starts Docker Compose services
- Creates a systemd service for automatic startup
- Configures PM2 for process management
- Automatic restart on failure
## Requirements
- Git (for cloning repository)
- Yarn (for dependency management)
- Docker and Docker Compose
- PM2 (for process management)
- Node.js/NVM installed
- SSH key access to the repository (if using private repos)
## Configuration Parameters
### Required
- **USER**: System user to run the application (default: `teclabmin`)
- **REPO**: Git repository URL (default: `git@github.com:bennidi/teclab-flintstone.git`)
- **APP**: Application name/directory name (default: `flintstone`)
### Optional
- **ENV**: Application environment (default: `production`)
- **AUTOSTART**: Enable and start service immediately (default: `True`)
- **NODE_PATH**: Path to Node.js binaries (default: `/home/teclabmin/.nvm/versions/node/v21.7.3/bin`)
## Example Usage
### Basic Configuration
```json
{
"USER": "myuser",
"REPO": "git@github.com:myorg/myapp.git",
"APP": "myapp"
}
```
### Advanced Configuration
```json
{
"USER": "appuser",
"REPO": "git@github.com:myorg/myapp.git",
"APP": "myapp",
"ENV": "staging",
"AUTOSTART": false,
"NODE_PATH": "/home/appuser/.nvm/versions/node/v20.0.0/bin"
}
```
## What This Cube Does
1. **Clone Repository**: Clones the specified Git repository to `/home/<USER>/<APP>`
2. **Install Dependencies**: Runs `yarn install` to install all dependencies
3. **Build Application**: Runs `yarn build` to compile the application
4. **Start Docker Services**: Runs `docker compose up -d` to start containerized services
5. **Create Startup Script**: Creates `/home/<USER>/<APP>.service.sh` that:
- Starts Docker Compose services
- Starts PM2 with ecosystem.config.js
6. **Create Systemd Service**: Creates `/etc/systemd/system/<APP>.service` that:
- Runs after Docker service
- Uses the specified user
- Configures proper environment (HOME, PATH)
- Auto-restarts on failure
7. **Enable & Start Service**: Enables and starts the service (if AUTOSTART=True)
## Service Management
### Check service status
```bash
sudo systemctl status <APP>
```
### Start the service
```bash
sudo systemctl start <APP>
```
### Stop the service
```bash
sudo systemctl stop <APP>
```
### Restart the service
```bash
sudo systemctl restart <APP>
```
### View service logs
```bash
sudo journalctl -u <APP> -f
```
### Disable autostart
```bash
sudo systemctl disable <APP>
```
## File Structure
After deployment:
```
/home/<USER>/
├── <APP>/ # Application directory
│ ├── ecosystem.config.js # PM2 configuration
│ ├── docker-compose.yml # Docker services
│ └── ... # Application files
├── <APP>.service.sh # Startup script
/etc/systemd/system/
└── <APP>.service # Systemd service file
```
## Troubleshooting
### Service fails to start
1. Check service logs:
```bash
sudo journalctl -u <APP> -n 50
```
2. Verify Docker is running:
```bash
sudo systemctl status docker
```
3. Check if Node.js path is correct:
```bash
which node
which pm2
```
### Repository clone fails
- Ensure SSH keys are properly configured for the user
- Test SSH access: `ssh -T git@github.com`
- Check repository URL is correct
### Docker Compose fails
- Verify Docker is installed and running
- Check docker-compose.yml exists in the application directory
- Ensure user has Docker permissions: `sudo usermod -aG docker <USER>`
### PM2 not starting
- Verify PM2 is installed: `pm2 --version`
- Check ecosystem.config.js exists
- Verify NODE_PATH includes PM2 binary location
## Notes
- The service type is set to `forking` to support PM2's daemon mode
- Service will auto-restart on failure with a 5-second delay
- Maximum 5 restart attempts in the burst period
- The service waits for Docker to be ready before starting
- Environment variables can be configured in the ecosystem.config.js file
packages/cubes-core/cubes/service/autostart/deploy.py
+27
View File
@@ -0,0 +1,27 @@
from pyinfra.operations import systemd
from pyinfra import host
APP = host.data.APP
# Enable and start the service based on AUTOSTART flag
if AUTOSTART:
systemd.service(
name=f'Enable {SERVICE_NAME} service',
service=APP,
enabled=True,
_sudo=True
)
systemd.service(
name=f'Start {SERVICE_NAME} service',
service=APP,
running=True,
_sudo=True
)
else:
server.shell(
name=f'Service {SERVICE_NAME} created but not enabled (AUTOSTART=False)',
commands=[f'echo "Service {SERVICE_NAME} is ready but not started. Enable with: sudo systemctl enable {APP} && sudo systemctl start {APP}"'],
_sudo=False
)
packages/cubes-core/cubes/service/autostart/manifest.mjs
+17
View File
@@ -0,0 +1,17 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'service:autostart',
name: 'Manage systemd service autostart',
dependencies: () => [],
schema: z.object({
APP: z.string().describe('The name of the systemd service (e.g., flintstone)'),
SERVICE_NAME: z
.string()
.optional()
.describe('Display name for the service')
.default('Application'),
AUTOSTART: z.boolean().describe('Should the service be enabled and started?').default(true),
}),
});
packages/cubes-core/cubes/ssh/authorize/README.md
+32
View File
@@ -0,0 +1,32 @@
# ssh-authorize
**Authorize SSH public key for a user**
## Purpose
This cube adds a specific SSH public key to a user's `authorized_keys` file on the remote server, allowing them to log in via SSH using the corresponding private key.
## Configuration
### Parameters
- **USER** (string, default: `'vagrant'`)
- The username on the remote server to authorize.
- If the user does not exist, `pyinfra` will attempt to create it (though a full user creation with shell/groups is better handled by `user-add`).
- **PUBKEY** (string, required)
- The actual content of the public key (e.g., `ssh-ed25519 AAAA...`).
## Use Cases
Grant access to a developer:
```javascript
exec('ssh-authorize', {
USER: 'developer',
PUBKEY: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...'
})
```
## Dependencies
None.
packages/cubes-core/cubes/ssh/authorize/deploy.py
+13
View File
@@ -0,0 +1,13 @@
from pyinfra import host
from pyinfra.operations import server
USER = host.data.USER
PUBKEY = host.data.PUBKEY
server.user(
name=f"Authorize public key for {USER}",
user=USER,
public_keys=[PUBKEY],
present=True,
_sudo=True
)
packages/cubes-core/cubes/ssh/authorize/manifest.mjs
+12
View File
@@ -0,0 +1,12 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'ssh:authorize',
name: 'Authorize SSH public key for a user',
dependencies: () => [],
schema: z.object({
USER: z.string().describe('Username to authorize').default('vagrant'),
PUBKEY: z.string().describe('SSH public key string').default(''),
}),
});
packages/cubes-core/cubes/ssh/keygen/README.md
+93
View File
@@ -0,0 +1,93 @@
# ssh-keygen
**Generate SSH key for a given user**
## Purpose
This cube generates a new SSH key pair for a specified user, which can be used for secure, passwordless authentication to remote servers and services like GitHub, GitLab, or other SSH-accessible systems.
## What are SSH Keys?
SSH keys provide a more secure and convenient way to authenticate compared to passwords:
- **Public key**: Shared with servers/services you want to access (like GitHub)
- **Private key**: Kept secret on your local machine, never shared
- **Passphrase-free**: This cube generates keys without a passphrase for automation
- **Algorithm support**: RSA (traditional) or Ed25519 (modern, recommended)
## What This Cube Does
1. Creates the `.ssh` directory with proper permissions (700)
2. Generates an SSH key pair using the specified algorithm
3. Saves the keys as `id_{SUFFIX}` and `id_{SUFFIX}.pub`
4. Logs the public key to the console for easy copying
## Configuration
### Parameters
- **SUFFIX** (string, default: `'ed25519'`)
- Suffix for keyname (e.g., `github` → `id_github.pub`)
- Helps identify the purpose of the key
- **EMAIL** (string, default: `'undefined@bitsquare.dev'`)
- Email address to associate with the SSH key
- Used as a comment in the public key
- **ALGORITHM** (enum: `'rsa'` | `'ed25519'`, default: `'ed25519'`)
- SSH key algorithm type
- **Ed25519**: Modern, faster, more secure (recommended)
- **RSA**: Traditional, widely supported
- **USER** (string, default: `'vagrant'`)
- Username for which to generate the SSH key
- Inherited from `user-add` dependency
## Dependencies
- **user-add** - Creates the user account first
## Post-Installation
After the key is generated:
1. The public key will be logged to the console
2. Copy the public key and add it to the target service:
- **GitHub**: Settings → SSH and GPG keys → New SSH key
- **GitLab**: Preferences → SSH Keys
- **Remote server**: Add to `~/.ssh/authorized_keys`
3. SSH config is automatically set up for the key
## Key Locations
- Private key: `/home/{USER}/.ssh/id_{SUFFIX}`
- Public key: `/home/{USER}/.ssh/id_{SUFFIX}.pub`
## Algorithm Comparison
**Ed25519** (Recommended):
- Smaller keys (256-bit)
- Faster generation and verification
- More secure against certain attacks
- Not supported on very old systems
**RSA**:
- Larger keys (2048-4096 bit)
- Universally supported
- Slower than Ed25519
- Well-tested and trusted
## Example Usage
Generate a key for GitHub access:
```javascript
exec('ssh-keygen', {
SUFFIX: 'github',
EMAIL: 'myemail@example.com',
ALGORITHM: 'ed25519',
USER: 'myuser'
})
```
This creates `id_github` and `id_github.pub` in `/home/myuser/.ssh/`.
packages/cubes-core/cubes/ssh/keygen/deploy.py
+46
View File
@@ -0,0 +1,46 @@
from pyinfra.operations import files, server, python
from pyinfra import host, logger
import logging
NAME=host.data.SUFFIX
EMAIL=host.data.EMAIL
ALGORITHM=host.data.ALGORITHM
USER=host.data.USER
# Ensure the .ssh directory exists
files.directory(
name="Ensure .ssh directory exists",
path=f"/home/{USER}/.ssh",
present=True,
mode=700,
user=USER,
group=USER,
)
# Generate the SSH keypair
server.shell(
name=f"Generate SSH key id_{NAME}",
commands=[
f"ssh-keygen -t {ALGORITHM} -f /home/{USER}/.ssh/id_{NAME} -C '{EMAIL}' -N ''"
]
)
# Print the public key
result = server.shell(
name="Print public key",
commands=[f"cat /home/{USER}/.ssh/id_{NAME}.pub"],
)
def callback():
# 🔹 Extract and log the key output
if result.stdout:
logger.info(f"Public Key for {USER}: {result.stdout.strip()}")
else:
logger.warning(f"No public key found for {USER} at /home/{USER}/.ssh/id_{NAME}.pub")
python.call(
name="Log public key",
function=callback,
)
packages/cubes-core/cubes/ssh/keygen/manifest.mjs
+20
View File
@@ -0,0 +1,20 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'ssh:keygen',
name: 'Generate SSH key for a given $USER',
dependencies: () => ['user:add'],
schema: z.object({
SUFFIX: z
.string()
.describe('Suffix for keyname, e.g. github => id_github.pub')
.default('ed25519'),
EMAIL: z
.string()
.describe('Email address to associate with the SSH key')
.default('undefined@bitsquare.dev'),
ALGORITHM: z.enum(['rsa', 'ed25519']).describe('SSH key algorithm type').default('ed25519'),
USER: z.string().describe('Username for which to generate the SSH key').default('vagrant'),
}),
});
packages/cubes-core/cubes/ssh/keyman/README.md
+110
View File
@@ -0,0 +1,110 @@
# ssh-keyman
**Deploy existing SSH keys to host**
## Purpose
This cube deploys pre-existing SSH key pairs from your local machine to a remote server, configuring them for automatic use with specified hosts (like GitHub, GitLab, etc.).
## What This Cube Does
1. **Copies SSH keys to the server**
- Transfers both private and public keys from local directory to remote `.ssh` folder
- Sets correct file permissions (600 for private, 644 for public)
2. **Configures SSH client**
- Creates/updates `.ssh/config` to use the deployed key for specified hosts
- Disables strict host key checking for easier automation
- Maps each host to use the correct identity file
3. **Ensures security**
- Sets proper directory permissions (700 for `.ssh`)
- Ensures keys are owned by the specified user
## Configuration
### Parameters
- **KEY_NAME** (string, default: `'id_ed25519'`)
- Name of the SSH key file (without extension)
- Must exist in the KEY_DIR directory locally
- **USER** (string, default: `'vagrant'`)
- Username for which to deploy the SSH key
- Inherited from `user-add` dependency
- **HOSTS** (string, default: `'github.com'`)
- Space-separated list of hosts to add to known_hosts
- Example: `'github.com gitlab.com bitbucket.org'`
## Dependencies
- **user-add** - Creates the user account first
## Use Cases
Deploy GitHub SSH key:
```javascript
exec('ssh-keyman', {
KEY_NAME: 'id_github',
HOSTS: 'github.com'
})
```
Deploy key for multiple Git services:
```javascript
exec('ssh-keyman', {
KEY_NAME: 'id_git',
HOSTS: 'github.com gitlab.com bitbucket.org'
})
```
## What Gets Configured
After deployment, the `.ssh/config` file will contain entries like:
```
Host github.com
IdentityFile /home/{USER}/.ssh/id_github
StrictHostKeyChecking no
```
This means when you run `git clone git@github.com:user/repo.git`, it will automatically use the deployed key.
## Key File Requirements
The local KEY_DIR must contain:
- `{KEY_NAME}` - Private key file
- `{KEY_NAME}.pub` - Public key file
For example, if `KEY_NAME=id_github`, you need:
- `./vault/tmp/id_github`
- `./vault/tmp/id_github.pub`
## Security Considerations
- **Private keys are sensitive**: Ensure your local KEY_DIR is secure
- **StrictHostKeyChecking disabled**: Convenient but less secure
- Consider enabling it for production: Edit `/home/{USER}/.ssh/config`
- **Backup your keys**: Keep secure copies of private keys
- **Use different keys**: Consider separate keys for different services
## Post-Installation
Test SSH connection:
```bash
ssh -T git@github.com
# Should show: "Hi username! You've successfully authenticated..."
```
Clone a repository:
```bash
git clone git@github.com:user/repo.git
# Should work without prompting for credentials
```
packages/cubes-core/cubes/ssh/keyman/deploy.py
+96
View File
@@ -0,0 +1,96 @@
from pyinfra.operations import server, files, apt, systemd
from pyinfra import host
import subprocess
import json
def get_keyman_config():
"""Get keyman configuration by calling keyman --print-config"""
try:
result = subprocess.run(
['keyman', '--print-config'],
capture_output=True,
text=True,
check=True
)
return json.loads(result.stdout)
except (subprocess.CalledProcessError, json.JSONDecodeError, FileNotFoundError):
return None
# Load keyman config for defaults
keyman_config = get_keyman_config()
USER = host.data.USER
KEY = host.data.KEY_NAME
# Use KEY_DIR from host data, or fall back to keyman config tmpDir
DIR = host.data.get('KEY_DIR')
if not DIR and keyman_config:
DIR = keyman_config.get('tmpDir')
if not DIR:
DIR = '../../vault/tmp' # Final fallback
# Support multiple hosts separated by space
HOSTS = map(str.lstrip, str(host.data.HOSTS).split(' '))
# 🔹 Define remote paths
SSH_DIR = f"/home/{USER}/.ssh" if USER != "root" else "/root/.ssh"
PRIVATE_KEY_PATH = f"{SSH_DIR}/{KEY}"
PUBLIC_KEY_PATH = f"{SSH_DIR}/{KEY}.pub"
# Ensure the .ssh directory exists
files.directory(
name="Ensure .ssh directory exists",
path=SSH_DIR,
present=True,
mode=700,
user=USER,
group=USER,
_sudo=True
)
# Copy private key to the remote server
files.put(
name="Copy private key",
src=f"{DIR}/{KEY}",
dest=PRIVATE_KEY_PATH,
mode="600",
user=USER,
group=USER,
_sudo=True,
)
# Copy public key to the remote server
files.put(
name="Copy public key",
src=f"{DIR}/{KEY}.pub",
dest=PUBLIC_KEY_PATH,
mode="644",
user=USER,
group=USER,
_sudo=True,
)
# Ensure correct permissions for the private key
server.shell(
name="Set correct permissions for private key",
commands=[f"chmod 600 {PRIVATE_KEY_PATH}"],
_sudo=True,
)
files.file(
name="Ensure .ssh/config directory exists",
path=f"/home/{USER}/.ssh/config",
present=True,
user=USER,
group=USER,
_sudo=True
)
for host in HOSTS:
files.line(
name=f"Configure SSH key for {host}",
path=f"{SSH_DIR}/config",
line=f"Host {host}\n IdentityFile {SSH_DIR}/{KEY}\n StrictHostKeyChecking no",
_sudo=True
)
packages/cubes-core/cubes/ssh/keyman/manifest.mjs
+19
View File
@@ -0,0 +1,19 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'ssh:keyman',
name: 'Deploy an ssh key managed by keyman',
dependencies: () => [],
schema: z.object({
KEY_NAME: z
.string()
.describe('Name of the SSH key file (without extension)')
.default('id_ed25519'),
USER: z.string().describe('Username for which to deploy the SSH key').default('vagrant'),
HOSTS: z
.string()
.describe('Space-separated list of hosts to add to known_hosts')
.default('github.com'),
}),
});
packages/cubes-core/cubes/user/add/README.md
+143
View File
@@ -0,0 +1,143 @@
# user-add
**Add a user with Fish shell and tools**
## Purpose
This cube creates a new user account with a modern shell environment (Fish), SSH key authentication, and enhanced productivity tools pre-configured.
## What This Cube Does
1. **Creates a new user account**
- Sets up home directory with proper permissions
- Configures password authentication
- Adds user to specified groups (e.g., `docker`, `sudo`)
- Sets Fish as the default login shell
2. **Configures SSH access**
- Deploys the specified SSH public key for passwordless authentication
- Creates `.ssh` directory with proper permissions
- Sets up SSH config file
- Configures SSH agent auto-loading for Fish shell
3. **Installs Fish shell enhancements**
- Installs **Oh My Fish** (OMF) - Fish shell framework with themes and plugins
- Deploys custom Fish configuration (`config.fish`)
- Sets up Fish rc directory for modular configurations
4. **Creates workspace directories**
- Creates `/home/{USER}/tmp` directory for temporary files
## Configuration
### Parameters
- **USER** (string, auto-generated)
- Username for the new user account
- Default: `userXXXXX` (randomly generated 5-character suffix)
- **PASSWORD** (string, **secret**)
- Password for the new user account
- Default: the literal `changeme` — a placeholder, not a credential. Change it
on first login, or pass a real one.
- Declared in the manifest's `secrets`, so it is never written to a session or
history file and is masked in printed commands. A replay asks for it again.
- It used to default to a randomly generated password. That was removed: since
the value is not recorded, an unattended run created an account with a
credential nobody had seen, and replaying that run produced a different one.
- **GROUPS** (string, default: `''`)
- Comma-separated list of additional groups (e.g., `"docker,sudo"`)
- Common groups:
- `docker` - Run Docker without sudo
- `sudo` - Administrative privileges
- `www-data` - Web server file access
- **PUBKEY** (string, **required** — no default)
- SSH public key to authorize for the user
- Should be your public key for passwordless SSH access
- There is deliberately no default. It used to be a specific personal key, so
accepting the default authorized *someone else's* key on the new account.
No key would be a sensible guess, so the cube asks instead.
- Because it is required, `--use-defaults` refuses to run this cube unless
`PUBKEY` comes from `env` in `.nopyrc.json`, a dependency, or a hook.
- Submitting an empty value at the prompt authorizes no key at all (the account
is still created, with password login only).
## Dependencies
- **apt:essentials** - Provides Fish shell and basic tools
## What is Fish?
Fish (Friendly Interactive Shell) is a modern command-line shell that focuses on usability:
- **Smart autosuggestions**: Suggests commands as you type based on history
- **Syntax highlighting**: Color-codes commands in real-time
- **Tab completions**: Comprehensive, discoverable command completions
- **No configuration needed**: Works great out of the box
## Post-Installation
After deployment:
- SSH into the server as the new user: `ssh {USER}@server`
- Your SSH key will be pre-authorized (no password needed if using key)
- Fish shell will start automatically with OMF installed
- SSH agent auto-loads to manage your SSH keys
## Notes
- The user's home directory is created at `/home/{USER}`
- Fish configuration is stored in `/home/{USER}/.config/fish/`
- Oh My Fish provides package management: `omf install <package>`
- To switch shells: `chsh -s /bin/bash` (or back to fish: `chsh -s /usr/bin/fish`)
---
# 📌 Most Useful Fish Key Bindings (with Fisher Extensions)
## 🐟 Default Fish Key Bindings
- `Ctrl + C` → Cancel the current command
- `Ctrl + D` → Exit the shell (or logout if in SSH)
- `Ctrl + L` → Clear the terminal
- `Ctrl + R` → Search command history (enhanced by `fzf.fish`)
- `Ctrl + U` → Delete the entire command line
- `Ctrl + W` → Delete the last word
- `Alt + ← / →` → Move backward/forward by a word
## 🔍 Enhanced with `fzf.fish`
- `Ctrl + R` → **Fuzzy search command history**
- `Ctrl + T` → **Fuzzy search and insert file path**
- `Alt + C` → **Fuzzy search directories (`cd` with `z`)**
## 📂 Directory Navigation (with `z`)
- `z <dir>` → Jump to a frequently used directory
- `z -l` → List most-used directories
- `z -c` → Remove a directory from `z`'s database
## 🔄 Process & Job Management
- `Ctrl + Z` → Suspend the current process
- `fg` → Bring a suspended process back to foreground
- `jobs` → List background jobs
## 🎨 Other Handy Shortcuts
- `fish_vi_key_bindings` → Enable Vi mode (press `Esc` for normal mode)
- `Ctrl + G` → Show Git status (if using `fzf.fish`)
- `Ctrl + E` → Edit command line in `$EDITOR`
## ⚙️ Useful Commands for Key Binding
```fish
# Set Fish default key bindings
fish_default_key_bindings
# Enable Vi mode
fish_vi_key_bindings
# Rebind a custom key (Example: Ctrl + G for git status)
bind \cg 'git status'
packages/cubes-core/cubes/user/add/config.fish
+10
View File
@@ -0,0 +1,10 @@
if status is-interactive
# Commands to run in interactive sessions can go here
# Execute all scripts in ~/.config/fish/rc/ on shell startup
for script in ~/.config/fish/rc/*.fish
if test -f $script
source $script
end
end
end
packages/cubes-core/cubes/user/add/deploy.py
+93
View File
@@ -0,0 +1,93 @@
from pyinfra import host
from pyinfra.operations import server, files, apt
from io import StringIO
# Define the username, password, and public key for the new admin user
USER = host.data.USER
HOME_DIR = f"/home/{USER}"
TMP_DIR = f"{HOME_DIR}/tmp"
PASSWORD = host.data.PASSWORD
# An empty submission at the prompt must not become an empty authorized_keys
# line, so an absent key means no key rather than a blank one.
PUBKEY = host.data.PUBKEY
PUBKEYS = [PUBKEY] if PUBKEY and str(PUBKEY).strip() else []
GROUPS = list(filter(None, map(str.strip, str(host.data.GROUPS).split())))
FISH_PATH = "/usr/bin/fish"
FISH_CONFIG_DIR = f"{HOME_DIR}/.config/fish"
FISH_CONFIG_FILE = f"{FISH_CONFIG_DIR}/config.fish"
FISH_RC_DIR = f"{FISH_CONFIG_DIR}/rc"
SSH_AGENT_SCRIPT = f"{FISH_RC_DIR}/ssh-agent.fish"
apt.packages(
name='Ensure fish shell is installed',
packages=[ 'fish'],
_sudo=True
)
# Ensure the user exists with a login shell
server.user(
name=f"Create user {USER} [{GROUPS}]",
present=True,
user=USER,
password=PASSWORD,
create_home=True,
groups=GROUPS,
shell=FISH_PATH,
public_keys=PUBKEYS,
_sudo=True
)
for dir in [f"{HOME_DIR}/.ssh", FISH_RC_DIR, TMP_DIR]:
files.directory(
name=f"Ensure {dir} directory exists",
path=dir,
present=True,
mode=700,
user=USER,
group=USER,
_sudo=True,
_sudo_user=USER,
_use_sudo_login=True
)
files.file(
name="Ensure .ssh/config exists",
path=f"{HOME_DIR}/.ssh/config",
present=True,
user=USER,
group=USER,
_sudo=True
)
server.shell(
name=f"Install OMF(Oh My Fish) for {USER}",
commands=[
f"curl https://raw.githubusercontent.com/oh-my-fish/oh-my-fish/master/bin/install > install-omf",
f"fish install-omf --yes --noninteractive",
],
_sudo=True,
_sudo_user=USER,
_use_sudo_login=True
)
files.put(
name="Add SSH agent auto-load script to Fish rc directory",
src="ssh-agent.fish",
dest=SSH_AGENT_SCRIPT,
user=USER,
group=USER,
mode="755", # Make it executable
_sudo=True,
)
files.put(
name="Add custom config.fish",
src="config.fish",
dest=FISH_CONFIG_FILE,
user=USER,
group=USER,
mode="755", # Make it executable
_sudo=True,
)
packages/cubes-core/cubes/user/add/manifest.mjs
+28
View File
@@ -0,0 +1,28 @@
import { Manifest, uniqid } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'user:add',
name: 'Add a user with fish shell and tools',
dependencies: () => ['apt:essentials'],
secrets: ['PASSWORD'],
schema: z.object({
USER: z
.string()
.describe('Username for the new user account')
.default(() => `user${uniqid(5)}`),
// A fixed placeholder, not a generated one: the password is never recorded
// in a session, so a generated default meant every run produced credentials
// nobody had seen and a replay produced different ones again.
PASSWORD: z.string().describe('Password for the new user account').default('changeme'),
GROUPS: z
.string()
.describe('Comma-separated list of additional groups (e.g., "docker,sudo")')
.default(''),
// No default on purpose. This used to carry a specific personal key, which
// meant an unattended run authorised someone else's key on the new account.
// Leaving it required makes `--use-defaults` refuse by name instead of
// guessing, and there is no key that would be a sensible guess.
PUBKEY: z.string().describe('SSH public key to authorize for the user'),
}),
});
packages/cubes-core/cubes/user/add/ssh-agent.fish
+14
View File
@@ -0,0 +1,14 @@
# Start SSH agent if not already running
if not set -q SSH_AUTH_SOCK
eval (ssh-agent -c)
end
# Add all private SSH keys in ~/.ssh to the agent
for key in ~/.ssh/id_*;
if test -f $key; and not string match -q "*pub" $key
ssh-add $key 2>/dev/null
end
end
# Export user and group ID for Docker
set -x UID (id -u)
set -x GID (id -g)
packages/cubes-core/cubes/user/edit/README.md
+52
View File
@@ -0,0 +1,52 @@
# user:edit
**Modify an existing user's password or group membership**
## Purpose
This cube allows you to update existing user accounts on the target system. It can be used to change passwords, add users to new groups (like `docker` or `sudo`), or revoke group memberships.
## What This Cube Does
1. Identifies the existing user on the target system
2. Updates the user's password if `PASSWORD` is provided
3. Adds the user to the groups specified in `GROUPS`
4. Removes the user from the groups specified in `GROUPS_ABSENT`
## Configuration
| Variable | Type | Description | Required |
| :--- | :--- | :--- | :--- |
| `USER` | `string` | The username of the account to modify | Yes |
| `PASSWORD` | `string` | New password for the user. **Secret**: never recorded in a session or history file, masked in printed commands, re-prompted on replay. | No |
| `GROUPS` | `string` | Comma-separated list of groups to ADD (e.g., `docker,sudo`) | No |
| `GROUPS_ABSENT` | `string` | Comma-separated list of groups to REMOVE | No |
## Dependencies
- `apt/essentials`: Standard system utilities.
## Usage
### Changing a Password
```bash
nopy install user:edit --env USER=myuser --env PASSWORD=newsecurepassword
```
### Adding a User to the Docker Group
```bash
nopy install user:edit --env USER=myuser --env GROUPS=docker
```
### Revoking Sudo Access
```bash
nopy install user:edit --env USER=myuser --env GROUPS_ABSENT=sudo
```
## Security Notes
- When setting passwords via the CLI, they may be visible in your shell history. Consider using a session file or interactive prompts for sensitive values.
- Changing your own user's groups or password may require a re-login to take full effect.
packages/cubes-core/cubes/user/edit/deploy.py
+24
View File
@@ -0,0 +1,24 @@
from pyinfra import host
from pyinfra.operations import server
# [agnt://cogen/cogen/user-edit-2]{cartridge: "ansiblings/cubes", action: "generated", status: "generated"}
"""
Deployment script for user:edit.
Updates password and group membership for an existing user.
"""
USER = host.data.USER
PASSWORD = host.data.get('PASSWORD')
GROUPS = [g.strip() for g in str(host.data.get('GROUPS', '')).split(',') if g.strip()]
GROUPS_ABSENT = [g.strip() for g in str(host.data.get('GROUPS_ABSENT', '')).split(',') if g.strip()]
# Update user details
server.user(
name=f"Update user {USER}",
user=USER,
password=PASSWORD,
groups=GROUPS,
groups_absent=GROUPS_ABSENT,
_sudo=True,
)
packages/cubes-core/cubes/user/edit/manifest.mjs
+27
View File
@@ -0,0 +1,27 @@
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
// [agnt://cogen/cogen/user-edit-1]{cartridge: "ansiblings/cubes", action: "generated", status: "generated"}
/**
* Manifest for the user:edit cube.
* Allows modifying existing user accounts (password, groups).
*/
export default Manifest({
id: 'user:edit',
name: 'user:edit - Modify an existing user account',
dependencies: () => [],
secrets: ['PASSWORD'],
schema: z.object({
USER: z.string().describe('The username of the account to modify'),
PASSWORD: z.string().optional().describe('New password for the user (optional)'),
GROUPS: z
.string()
.optional()
.describe('Comma-separated list of groups the user SHOULD be in (optional)'),
GROUPS_ABSENT: z
.string()
.optional()
.describe('Comma-separated list of groups to REMOVE from the user (optional)'),
}),
});
packages/cubes-core/package.json
+45
View File
@@ -0,0 +1,45 @@
{
"name": "@bitsquare/cubes-core",
"version": "1.0.0-alpha0",
"description": "The core nopy cube bundle: apt, users, ssh, networking, services and runtimes.",
"keywords": [
"nopy",
"nopy-cubes",
"pyinfra",
"deployment",
"infrastructure"
],
"license": "MIT",
"author": "bitsquare",
"type": "module",
"repository": {
"type": "git",
"url": "git+https://gitea.bitsquare.dev/BitSquare/ansiblings.git",
"directory": "packages/cubes-core"
},
"homepage": "https://gitea.bitsquare.dev/BitSquare/ansiblings/src/branch/main/packages/cubes-core",
"bugs": {
"url": "https://gitea.bitsquare.dev/BitSquare/ansiblings/issues"
},
"engines": {
"node": ">=22"
},
"nopy": {
"cubes": [
"./cubes"
]
},
"files": [
"cubes",
"!cubes/**/*.log",
"README.md",
"LICENSE"
],
"publishConfig": {
"access": "public"
},
"dependencies": {
"@bitsquare/nopy-cube": "workspace:*",
"zod": "^4.4.3"
}
}
packages/nopy-cube/LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 bitsquare
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
packages/nopy-cube/README.md
+69
View File
@@ -0,0 +1,69 @@
# @bitsquare/nopy-cube
The authoring surface for [nopy](https://www.npmjs.com/package/@bitsquare/nopy)
cubes — the `Manifest` factory, the `Cube` class, and the types around them.
A cube manifest ships nothing but data, so it should not have to depend on a CLI
to describe itself. This package is what a **cube bundle** depends on: no
`commander`, no `inquirer`, no `execa`, no process spawning. `@bitsquare/nopy`
re-exports everything here, so a manifest that already imports from
`@bitsquare/nopy` keeps working unchanged.
## Install
```sh
pnpm add @bitsquare/nopy-cube zod
```
`zod` is a **peer dependency** on purpose: the manifest, the schema it builds and
the `Manifest` factory should all see the same copy.
## Writing a manifest
```js
// cubes/net/tailscale/manifest.mjs
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'net:tailscale',
name: 'Tailscale',
schema: z.object({
AUTH_KEY: z.string().describe('Tailscale auth key'),
ACCEPT_ROUTES: z.boolean().describe('Accept advertised routes').default(true),
}),
secrets: ['AUTH_KEY'],
dependencies: (vars) => (vars.ACCEPT_ROUTES ? ['net:ip-forwarding'] : []),
before: [async (ctx, vars) => ctx.exec('apt:essentials', {})],
});
```
Every schema field should carry a `.describe()` — nopy uses it as the prompt
label — and a `.default()` wherever a sensible one exists, so `--use-defaults`
can run the cube without prompting.
`secrets` names the schema keys that hold sensitive values. Nopy keeps those out
of session and history files and masks them in every command it prints; it does
not infer them, so a key nothing declares is recorded and printed in the clear.
Each entry must be a key of `schema` — naming anything else is a manifest error.
Give a secret a placeholder `.default()` rather than a real credential: a default
lives in the manifest, where none of that protection reaches it.
The manifest lives next to a `deploy.py` in the same directory; together they
make a cube. See the
[nopy README](https://www.npmjs.com/package/@bitsquare/nopy) for the full cube
contract and for how to publish a directory of cubes as a bundle.
## Exports
| Export | What it is |
| ----------------------------------- | -------------------------------------------------------------- |
| `Manifest(opts)` | Builds a manifest, filling in `id`, `schema`, `secrets`, `before`, `after` |
| `createManifest` / `manifest` | Aliases of `Manifest` |
| `Cube` | A loaded manifest plus its directory; `getDefaults()`, `requiredKeys()`, `secrets`, `isSecret()` |
| `zodKind` / `zodInner` | Instance-agnostic zod introspection, safe across zod copies |
| `AnyObjectSchema`, `CubeVariables`, `DependencySpec`, `Hook`, `HookContext`, `CubeSource`, `LoadResult` | types |
## License
MIT
packages/nopy-cube/package.json
+60
View File
@@ -0,0 +1,60 @@
{
"name": "@bitsquare/nopy-cube",
"version": "1.0.0-alpha0",
"description": "Authoring types for nopy cubes: the Manifest factory and the Cube contract.",
"keywords": [
"nopy",
"pyinfra",
"deployment",
"infrastructure"
],
"license": "MIT",
"author": "bitsquare",
"type": "module",
"repository": {
"type": "git",
"url": "git+https://gitea.bitsquare.dev/BitSquare/ansiblings.git",
"directory": "packages/nopy-cube"
},
"homepage": "https://gitea.bitsquare.dev/BitSquare/ansiblings/src/branch/main/packages/nopy-cube",
"bugs": {
"url": "https://gitea.bitsquare.dev/BitSquare/ansiblings/issues"
},
"engines": {
"node": ">=22"
},
"exports": {
".": {
"types": "./dist/index.d.ts",
"default": "./dist/index.js"
},
"./package.json": "./package.json"
},
"files": [
"dist",
"README.md",
"LICENSE"
],
"publishConfig": {
"access": "public"
},
"scripts": {
"clean": "rm -rf dist .tsbuildinfo",
"build": "tsc",
"prepack": "pnpm run build",
"link:local": "pnpm run build && npm link",
"test": "vitest run",
"test:coverage": "vitest run --coverage",
"test:watch": "vitest"
},
"peerDependencies": {
"zod": "^4.4.3"
},
"devDependencies": {
"@types/node": "^26.1.1",
"@vitest/coverage-v8": "^4.1.10",
"typescript": "^7.0.2",
"vitest": "^4.1.10",
"zod": "^4.4.3"
}
}
packages/nopy/src/cubes/factories.ts → packages/nopy-cube/src/factories.ts
+1 -1
View File
@@ -1,6 +1,6 @@
/**
* Factory functions for creating cube configurations
* @module cubes/factories
* @module factories
*/
import { type AnyObjectSchema, Manifest } from './types.js';
packages/nopy-cube/src/index.ts
+31
View File
@@ -0,0 +1,31 @@
/**
* @bitsquare/nopy-cube — the authoring surface for nopy cubes.
*
* Everything a `manifest.mjs` needs and nothing else: no CLI, no prompts, no
* process spawning. `@bitsquare/nopy` re-exports all of it, so a manifest can
* import from either package.
*
* @packageDocumentation
*/
export {
createManifest,
ManifestFactory,
manifest,
} from './factories.js';
export type {
AnyObjectSchema,
CubeSource,
CubeVariables,
DependencySpec,
Hook,
HookContext,
LoadResult,
} from './types.js';
export {
Cube,
Manifest,
zodInner,
zodKind,
} from './types.js';
export { uniqid } from './utils.js';
packages/nopy/src/cubes/types.ts → packages/nopy-cube/src/types.ts
+23 -1
View File
@@ -1,6 +1,6 @@
/**
* Type definitions for Nopy cubes
* @module cubes/types
* @module types
*/
import { z } from 'zod';
@@ -47,6 +47,18 @@ export interface Manifest<Schema extends AnyObjectSchema = AnyObjectSchema> {
name: string;
/** Zod schema for validating cube variables */
schema: Schema;
/**
* Schema keys holding secrets. Their values are never written to a session
* file, and are masked wherever a command or a variable would be printed.
*
* A plain array rather than schema-level metadata on purpose: `.meta()` and
* `.describe()` both store into zod's global registry, which is per-copy — a
* manifest that builds its schema with its own zod writes the marker into a
* registry this process cannot read. A missed `.describe()` costs an ugly
* prompt label; a missed secret marker writes a password to disk, so this one
* cannot be allowed to fail open. See {@link zodKind} for the same hazard.
*/
secrets?: string[];
/** Dynamic dependency resolver based on collected variables */
dependencies?: (variables: z.infer<Schema>) => DependencySpec[];
/** Hooks to run before cube execution */
@@ -65,6 +77,7 @@ export function Manifest<Schema extends AnyObjectSchema>(
id: opts.id ?? '',
name: opts.name,
schema: opts.schema ?? (z.object({}) as unknown as Schema),
secrets: opts.secrets ?? [],
dependencies: opts.dependencies,
before: opts.before ?? [],
after: opts.after ?? [],
@@ -189,6 +202,15 @@ export class Cube<Schema extends AnyObjectSchema = AnyObjectSchema> {
.filter(([, zodType]) => !zodType.safeParse(undefined).success)
.map(([key]) => key);
}
/** Schema keys the manifest declared as secrets. */
get secrets(): string[] {
return this.manifest.secrets ?? [];
}
isSecret(key: string): boolean {
return this.secrets.includes(key);
}
}
/**
packages/nopy/src/cubes/utils.ts → packages/nopy-cube/src/utils.ts
+1 -1
View File
@@ -1,6 +1,6 @@
/**
* Utility functions for cubes
* @module cubes/utils
* @module utils
*/
/**
packages/nopy/tests/cubes.factories.test.ts → packages/nopy-cube/tests/factories.test.ts
+2 -2
View File
@@ -1,10 +1,10 @@
/**
* Tests for cubes/factories module
* Tests for the manifest factories
*/
import { describe, expect, it } from 'vitest';
import { z } from 'zod';
import { createManifest, manifest } from '../src/cubes/factories.js';
import { createManifest, manifest } from '../src/factories.js';
describe('createManifest', () => {
it('creates manifest with basic properties', () => {
packages/nopy-cube/tests/helpers/foreign-zod.ts
+29
View File
@@ -0,0 +1,29 @@
/**
* A schema that behaves like zod's but does not share zod's prototypes.
*
* Once cubes arrive from `node_modules`, the schema a manifest builds may come
* from a *second* copy of zod — its own dependency, or one shipped inside a
* bundle. Such a schema is structurally identical and `instanceof` blind to it.
* Rebuilding the nodes as plain objects reproduces that from inside a single
* process, so anything that reads zod's internals stays pinned to `def.type`.
*/
import type { z } from 'zod';
/** Strips the prototype off a schema node and everything it wraps. */
function strip(node: unknown): unknown {
const def = { ...(node as { def: Record<string, unknown> }).def };
if (def.innerType) def.innerType = strip(def.innerType);
return { def };
}
export function foreignZodSchema<S extends z.ZodObject<any>>(schema: S): S {
return {
// Parsing is not what is under test — delegate it and keep the real
// behaviour, so only the introspection path sees the foreign nodes.
safeParse: (value: unknown) => schema.safeParse(value),
shape: Object.fromEntries(
Object.entries(schema.shape).map(([key, node]) => [key, strip(node)])
),
} as unknown as S;
}
packages/nopy/tests/cubes.types.test.ts → packages/nopy-cube/tests/types.test.ts
+61 -1
View File
@@ -5,12 +5,39 @@
import { describe, expect, it } from 'vitest';
import { z } from 'zod';
import { Cube, Manifest } from '../src/cubes/types.js';
import { Cube, Manifest } from '../src/types.js';
import { foreignZodSchema } from './helpers/foreign-zod.js';
const cube = (schema: z.ZodObject<any>) =>
new Cube(Manifest.create({ id: 'c', name: 'C', schema }), '/cubes/c', 'deploy.py');
describe('Cube', () => {
it('reads id and name off the manifest', () => {
const c = cube(z.object({}));
expect(c.id).toBe('c');
expect(c.name).toBe('C');
});
it('defaults its source to its own directory', () => {
// What the loader overrides when a cube arrives from a package; a cube
// built by hand still has to answer the question.
expect(cube(z.object({})).source).toEqual({ type: 'dir', dir: '/cubes/c' });
});
it('keeps the source it was constructed with', () => {
const source = { type: 'package' as const, packageName: '@acme/cubes-net', dir: '/pkg/cubes' };
const c = new Cube(
Manifest.create({ id: 'c', name: 'C' }),
'/pkg/cubes/c',
'deploy.py',
source
);
expect(c.source).toBe(source);
});
});
describe('Cube.getDefaults', () => {
it('resolves every default when the whole schema parses', () => {
const c = cube(
@@ -109,3 +136,36 @@ describe('Cube.requiredKeys', () => {
expect(c.requiredKeys()).toEqual([]);
});
});
describe('Cube.secrets', () => {
it('is empty when the manifest declares none', () => {
const c = cube(z.object({ PASSWORD: z.string().default('x') }));
expect(c.secrets).toEqual([]);
// No name-based guessing: only what the manifest says.
expect(c.isSecret('PASSWORD')).toBe(false);
});
it('reports what the manifest declared', () => {
const c = new Cube(
Manifest.create({
id: 'c',
name: 'C',
schema: z.object({ USER: z.string(), PASSWORD: z.string() }),
secrets: ['PASSWORD'],
}),
'/cubes/c',
'deploy.py'
);
expect(c.secrets).toEqual(['PASSWORD']);
expect(c.isSecret('PASSWORD')).toBe(true);
expect(c.isSecret('USER')).toBe(false);
});
it('defaults to an empty list on a manifest built by hand', () => {
const c = new Cube({ id: 'c', name: 'C', schema: z.object({}) }, '/cubes/c', 'deploy.py');
expect(c.secrets).toEqual([]);
});
});
packages/nopy/tests/cubes.utils.test.ts → packages/nopy-cube/tests/utils.test.ts
+2 -2
View File
@@ -1,9 +1,9 @@
/**
* Tests for cubes/utils module
* Tests for the uniqid helper
*/
import { describe, expect, it } from 'vitest';
import { uniqid } from '../src/cubes/utils.js';
import { uniqid } from '../src/utils.js';
describe('uniqid', () => {
it('generates string of default length (5)', () => {
packages/nopy-cube/tsconfig.json
+15
View File
@@ -0,0 +1,15 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"tsBuildInfoFile": ".tsbuildinfo",
"outDir": "dist",
"rootDir": "src",
"lib": ["ES2020"],
"composite": true,
"module": "NodeNext",
"types": ["node"]
},
"include": ["src/**/*"],
"exclude": ["coverage", "node_modules", "dist"],
"references": []
}
packages/nopy-cube/vitest.config.ts
+25
View File
@@ -0,0 +1,25 @@
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
globals: true,
environment: 'node',
include: ['src/**/*.test.ts', 'tests/**/*.test.ts'],
coverage: {
provider: 'v8',
reporter: ['text', 'json-summary', 'html'],
include: ['src/**/*.ts'],
exclude: [
'src/**/*.test.ts',
// Pure re-export barrel: no logic to cover.
'src/index.ts',
],
thresholds: {
branches: 85,
functions: 85,
lines: 80,
statements: 80,
},
},
},
});
packages/nopy/README.md
+112 -17
View File
@@ -33,7 +33,7 @@ Nopy wraps pyinfra with structure, validation, and an interactive experience for
A cube is a **directory** containing two files:
- **JavaScript manifest**: `manifest.mjs` defining schema, dependencies, defaults, and hooks
- **JavaScript manifest**: `manifest.mjs` defining schema, dependencies, defaults, secrets, and hooks
- **Python deployment script**: `deploy.py`, a plain pyinfra script
Configuration variables are declared in the manifest and validated with Zod schemas before the deployment script runs.
@@ -96,7 +96,7 @@ apt.packages(
)
```
Every key defined in the manifest `schema` is guaranteed to be present on `host.data` — either from the Zod `.default()`, from `.nopyrc.json`, from a dependency, or from a user prompt.
Every key defined in the manifest `schema` is guaranteed to be present on `host.data` — either from the Zod `.default()`, from `.nopyrc.json`, from a recorded session, from a dependency, or from a user prompt.
**Value types**: pyinfra parses `--data` values before your script sees them. `"true"` / `"false"` become booleans, numeric strings become `int`, valid JSON becomes the parsed structure, and everything else stays a string. This is why `UPDATE` can be handed straight to pyinfra's `update=` argument, while `PACKAGES` is wrapped in `str(...)` before splitting.
@@ -104,18 +104,57 @@ Every key defined in the manifest `schema` is guaranteed to be present on `host.
Variable defaults are defined directly in the Zod schema using `.default()`. This ensures that every cube has a predictable starting state and provides type-safe default values.
**Priority order (lowest to highest):**
A variable can be set from several places in one run. Every assignment is kept, tagged with where it came from — its **origin** — and the highest-ranked origin wins.
1. Zod schema `.default()` values
2. Global `env` from `.nopyrc.json`
3. User prompts, or the recorded answers on session replay
4. Variables passed in by a dependency or a hook
**Origins, lowest to highest:**
| Origin | Set by |
| --------- | ------------------------------------------------------- |
| `default` | the Zod schema's `.default()` |
| `env` | the `env` block of `.nopyrc.json` |
| `session` | a value recorded in a session file or history entry |
| `prompt` | what the user typed |
| `param` | a dependency spec or a `before`/`after` hook |
This allows cubes to ship with reasonable defaults while still allowing users to override them globally via `.nopyrc.json` or interactively during deployment. Because `env` outranks the schema, `.nopyrc.json` is also what steers a run started with `--use-defaults`, which never prompts.
3 and 4 rarely compete: a key a dependency supplies is left out of the prompt entirely, so the user is only ever asked about the keys nothing else has set.
`prompt` and `param` rarely compete: a key a dependency supplies is left out of the prompt entirely, so the user is only ever asked about the keys nothing else has set.
A field declared without `.default()` has none of sources 1 and 2 to fall back on. It is prompted for like any other, with an empty initial value — but a run that cannot prompt (`--use-defaults`) fails on it unless `env` or a dependency provides it.
Ranking by origin rather than by arrival order is what makes replay work: a recorded value is applied *before* the cube would be prompted for, and prompting can still override it, but a `--data` value pushed in by a dependency is never clobbered by a stale recording.
A field declared without `.default()` has no `default` origin to fall back on. It is prompted for like any other, with an empty initial value — but a run that cannot prompt (`--use-defaults`) fails on it unless `env` or a dependency provides it.
#### Secrets
A manifest can name schema keys that hold sensitive values:
```javascript
export default cubes.Manifest({
id: 'user:add',
name: 'Add a user account',
secrets: ['PASSWORD'],
schema: z.object({
USERNAME: z.string().describe('Username for the new account').default('deploy'),
PASSWORD: z.string().describe('Password for the new user account').default('changeme'),
})
})
```
Every entry must be a key of `schema`; naming anything else is a manifest error and aborts the run, so a typo fails loudly instead of silently leaving a value unprotected.
Declaring a key a secret changes three things:
- **It is never written to a session file or to the history.** Everything else the run settled on is recorded — including values that came from a `.default()` — but declared secrets are left out.
- **It is masked wherever a command or a plan is printed** — `--dry-run`, `--print-only`, and the debug log all show `********` in place of the value, in the variable list *and* in the `pyinfra` command line above it. The SSH password passed via `--password` is masked the same way, whether or not any cube declares secrets.
- **It is re-prompted on replay**, since there is nothing recorded to replay from (see [Session Recording and Replay](#session-recording-and-replay)).
Nopy does not guess. A key called `PASSWORD` in a manifest that declares no `secrets` is treated as an ordinary variable — recorded, and printed in the clear.
Three limits are worth knowing, because `secrets` keeps a value out of the files nopy writes and nothing more:
- **It is on the command line.** pyinfra takes its data as `--data KEY=value`, so the real value is visible in `ps` for as long as the deployment runs. Masking covers nopy's own output, not the process table.
- **The prompt shows it.** The variable form displays and pre-fills what it is asking about, so a secret is on screen while it is being entered or confirmed.
- **A `.default()` is not protected.** A default lives in the manifest, in plain text, wherever the manifest is checked in. Give a secret a placeholder default like `changeme` if it needs one at all, never a real credential.
### Configuration
@@ -125,6 +164,7 @@ Uses `.nopyrc.json` files (project-level or home directory) containing:
{
"hosts": ["host1.example.com", "host2.example.com"],
"cubeDirs": ["./cubes", "../shared-cubes"],
"cubePackages": ["@bitsquare/cubes-core"],
"env": {
"SHARED_VAR": "value"
},
@@ -144,6 +184,8 @@ Uses `.nopyrc.json` files (project-level or home directory) containing:
`history` controls automatic session recording (see [Deployment History](#deployment-history)), and `execution.continueOnError` sets the default for `--continue-on-error`.
`cubeDirs` holds paths, `cubePackages` holds installed npm packages that ship cubes — see [Cube Discovery](#cube-discovery) below and [CUBE-BUNDLES.md](docs/CUBE-BUNDLES.md) for publishing your own. Both are additive, and both resolve relative to the config file that named them, not to the working directory: a `.nopyrc.json` two levels up may name a package that only exists in *its* `node_modules`.
#### Logging Configuration
Control pyinfra output verbosity and debug information using the `log` configuration object:
@@ -213,12 +255,16 @@ Sessions are stored in `.nopysession.json` files with the following structure:
**Structure Details:**
- **`cubes`**: Array of cubes with only cube-specific variables (not global env vars)
- **`env`**: Global environment variables shared across cubes (like in `.nopyrc.json`)
- **`cubes`**: Array of cubes with the variable values that cube ran with
- **`env`**: The `env` block of `.nopyrc.json` as it stood at record time, kept for reference
- **`hosts`**: Array of target hosts
- **`auth`**: Authentication configuration (passwords are never stored)
**Security Note**: Passwords are never stored in session files. If a session uses password authentication, you'll be prompted for the password during replay.
**What is recorded:** every value each cube settled on, regardless of where it came from — a value the user typed, one inherited from `.nopyrc.json` `env`, one a dependency supplied, and one that fell through to the schema's `.default()` are all written out the same way. A session is therefore a full snapshot rather than a diff, and a `--use-defaults` run produces a session with real values in it instead of an empty one.
The consequence is that replay is faithful rather than re-derived: the recorded value outranks the current `.nopyrc.json` `env` and the current schema default, so editing either one does not silently change what a replay does. To pick up a new default, record a fresh session.
**Security Note**: Passwords are never stored in session files. This covers both the SSH password — a session records the auth *method* and username, never the credential — and any schema key a cube's manifest lists under [`secrets`](#secrets). Both are re-prompted on replay.
#### Recording a Session
@@ -240,12 +286,48 @@ nopy install --load-session my-deployment.nopysession.json
# Only password authentication will prompt for credentials
```
A replay runs straight through without asking anything, with three exceptions. Password authentication always re-prompts. A session with no recorded host falls back to the host picker. And a cube is re-prompted for its declared secrets, plus for any required variable the session has no value for — which happens when the cube's schema has gained a field since the session was written.
Those re-prompts are what a session cannot supply, so `--use-defaults` cannot paper over them: combining `-D` with a replay that needs either fails with a message naming the keys rather than deploying with a placeholder. Put the values under `env` in `.nopyrc.json` to make such a replay unattended.
### Cube Discovery
Nopy searches for cubes in:
1. Directories specified in `.nopyrc.json` `cubeDirs`
2. Directories containing a `.npcubes` marker file (searching upwards from current directory)
2. Cube directories of every package listed in `.nopyrc.json` `cubePackages`
3. Directories containing a `.npcubes` marker file (searching upwards from current directory)
All three are unioned and scanned the same way. A directory is a cube when it holds both a manifest (`manifest.mjs` or `*.manifest.mjs`) and a deploy script (`deploy.py` or `*.deploy.py`); dotted directories and `node_modules` are skipped during the scan.
#### Cube packages
A cube package is an ordinary npm package that ships cube directories and points at them from its own `package.json`:
```json
{
"name": "@bitsquare/cubes-core",
"nopy": { "cubes": ["./cubes"] }
}
```
Install it and name it — nothing needs linking or copying:
```sh
pnpm add -D @bitsquare/cubes-core
```
```json
{ "cubePackages": ["@bitsquare/cubes-core"] }
```
Naming a package is a statement that cubes are expected from it, so anything wrong is an error that aborts the run rather than a silent skip: the package is not installed, it declares no `nopy.cubes`, or an entry points at a directory that does not exist or lies outside the package.
#### Ids are claimed globally
A cube id such as `apt:essentials` is claimed across every source at once, not per directory or per package. Two cubes with the same id abort the run with an error naming both and where each came from. There is no precedence rule and no shadowing — a local cube does not quietly win over a packaged one, in either direction. Prefix your own cubes distinctly if you point `cubeDirs` at a local tree alongside an installed bundle.
Writing cubes to publish is covered in [CUBE-BUNDLES.md](docs/CUBE-BUNDLES.md).
## Command Line Usage
@@ -319,6 +401,17 @@ have no default values. Set them under "env" in .nopyrc.json, pass them from a
dependency, or drop --use-defaults to be prompted.
```
Pairing `-D` with a replay fails the same way when the replay would have to ask
something — a declared [secret](#secrets), which is never recorded, or a required
variable the session has no value for. Both are the sources `-D` has no substitute
for, so it stops rather than deploying a placeholder:
```
Error: Cube "user:add" cannot be replayed with --use-defaults: PASSWORD would
have to be entered. Secrets are never recorded in a session. Replay without
--use-defaults, or set the values under "env" in .nopyrc.json.
```
**Use SSH key authentication**:
```bash
@@ -434,20 +527,20 @@ Session History:
Total: 2 session(s)
```
Each entry records the selected cubes together with the variable values that were answered at the prompts, the target hosts, the authentication method, and the username — never the password. Pass an ID to `-H` to run that exact combination again:
Each entry records the selected cubes together with every variable value they ran with, the target hosts, the authentication method, and the username — never the password, and never a key the manifest declared a [secret](#secrets). Pass an ID to `-H` to run that exact combination again:
```bash
nopy install -H mdk0zzp8b71cq
```
A replay is non-interactive: cube selection, host, and variable values all come from the entry, so nopy runs straight through without asking anything. The two exceptions are password authentication, which always re-prompts, and an entry with no recorded host, which falls back to the host picker.
A replay is non-interactive: cube selection, host, and variable values all come from the entry, so nopy runs straight through without asking anything. It asks only for what the entry cannot hold — the password under password authentication, and any declared secret — plus the host picker when the entry recorded none.
Two things are worth knowing before relying on an older entry:
- **Recorded values are applied as defaults, not as a frozen snapshot.** If a cube's schema has gained a variable since the run was recorded, the replay neither prompts for it nor fails — the new variable quietly takes its Zod `.default()`. Global `env` values are likewise read from the *current* `.nopyrc.json` rather than from the entry.
- **Recorded values win over the current configuration.** The entry is a snapshot of everything the run settled on, so editing a cube's `.default()` or the `env` block of `.nopyrc.json` afterwards does not change what the replay does. A variable the schema has gained *since* the entry was written has nothing recorded: if it has a `.default()` the replay quietly takes it, and if it is required the replay prompts for it.
- **A replay fails if a cube no longer exists.** Renaming or deleting a cube id makes every history entry that referenced it unreplayable: nopy logs `Cube from session not found` and then aborts with `Cube not found: <id>`.
The history lives in `.nopy.history.json` in the working directory and uses the same structure as a session file, so trimming the array by hand is a perfectly good way to prune it. It does contain the variable values that were entered, which is why it is listed in this repository's `.gitignore` — treat it like any other file holding deployment configuration. A corrupt or unreadable history file is treated as empty rather than raising an error, which looks exactly like a project that has never been deployed from.
The history lives in `.nopy.history.json` in the working directory and uses the same structure as a session file, so trimming the array by hand is a perfectly good way to prune it. It does contain the variable values a run used, which is why it is listed in this repository's `.gitignore` — treat it like any other file holding deployment configuration. A corrupt or unreadable history file is treated as empty rather than raising an error, which looks exactly like a project that has never been deployed from.
For a run you want to keep indefinitely, don't rely on history — it rotates. Use `--save-session` to write it to a file you control (see [Session Recording and Replay](#session-recording-and-replay)).
@@ -468,7 +561,9 @@ npm run debug
## Documentation
- [Cube Hooks](docs/HOOKS.md) - Lifecycle hooks for dynamic orchestration
- [Cube Bundles](docs/CUBE-BUNDLES.md) - Distributing cubes as npm packages
- [Session Format](docs/SESSION_FORMAT.md) - Internal JSON/MJS session structure
- [API Reference](docs/API.md) - Types and exported functions
## Resources
packages/nopy/docs/API.md
+71 -1
View File
@@ -65,6 +65,20 @@ interface NopyResult {
The cubes module provides types and functions for working with deployment units.
The authoring half of it — `Manifest`, `Cube`, `Hook`, `uniqid` and the rest —
actually lives in **[`@bitsquare/nopy-cube`](../../nopy-cube)**, a package with
no CLI and no dependency other than zod. `@bitsquare/nopy` re-exports all of it,
so both of these work:
```javascript
import { Manifest } from '@bitsquare/nopy-cube'; // in a manifest.mjs — prefer this
import { cubes } from '@bitsquare/nopy'; // cubes.Manifest — still supported
```
Import from `nopy-cube` in a cube bundle you intend to publish: it lets the
bundle depend on the authoring types without pulling the whole CLI in as a
dependency. See [CUBE-BUNDLES.md](CUBE-BUNDLES.md).
### Types
#### `Cube<Schema>`
@@ -76,6 +90,7 @@ interface Cube<Schema extends z.AnyZodObject = z.AnyZodObject> {
key: string; // Unique identifier
name: string; // Human-readable name
dir: string; // Absolute path to cube directory
source: CubeSource; // Where it was discovered
dependencies: string[];
schema: Schema;
defaults: () => z.infer<Schema>;
@@ -84,6 +99,18 @@ interface Cube<Schema extends z.AnyZodObject = z.AnyZodObject> {
}
```
#### `CubeSource`
Where a cube came from. Carried so that a duplicate-id error can name the origin
of each claimant, which is the difference between a usable error message and a
puzzle when the collision is between a local tree and an installed bundle.
```typescript
type CubeSource =
| { type: 'dir'; dir: string }
| { type: 'package'; packageName: string; dir: string };
```
#### `Manifest<Schema>`
Cube manifest (used in `manifest.mjs` files).
@@ -124,7 +151,9 @@ interface HookContext {
#### `loadCubes()`
Loads all cubes from discovered cube directories.
Loads all cubes from discovered cube directories — `cubeDirs`, the directories
declared by every package in `cubePackages`, and any ancestor directory holding a
`.npcubes` marker.
```typescript
const { cubes, errors } = await loadCubes();
@@ -139,6 +168,27 @@ interface LoadResult {
}
```
`errors` is non-empty for a duplicate id, a manifest that fails to load, a
package in `cubePackages` that is not installed or declares no cubes, and a
`nopy.cubes` entry that is missing or points outside its package. Any of them
aborts the run — none is a silent skip.
#### `resolveCubePackages(refs)`
Resolves `CubePackageRef[]` to installed packages and their cube directories.
Called by `loadCubes()`; exported because the resolution failures are worth
testing on their own.
```typescript
const { packages, errors } = resolveCubePackages(config.cubePackages);
interface CubePackage {
name: string; // the name it was requested under
root: string; // absolute path to the package root
dirs: string[]; // absolute paths from its `nopy.cubes` field
}
```
#### `resolveDependencies(cubes, selectedCubeNames)`
Resolves all transitive dependencies for selected cubes.
@@ -452,11 +502,31 @@ Configuration file structure.
interface NopyConfig {
hosts: string[];
cubeDirs: string[];
cubePackages: CubePackageRef[];
env: EnvConfig;
log?: LogConfig;
}
```
#### `CubePackageRef`
A package named in `cubePackages`, paired with where it was named. In the config
file an entry is just a string (`"@bitsquare/cubes-core"`); `loadConfig()`
normalises it.
```typescript
interface CubePackageRef {
/** The package name, as written in the config. */
spec: string;
/** Directory of the config file that named it — resolution starts here. */
from: string;
}
```
`from` is what makes a package named in a parent config resolve against *that*
config's `node_modules`, not the working directory's. It is the same problem
`PATH_PROPERTIES` solves for relative `cubeDirs`.
#### `LogConfig`
Logging configuration.
packages/nopy/docs/CUBE-BUNDLES.md
+263
View File
@@ -0,0 +1,263 @@
# Cube bundles
How to package cubes as an npm package so other projects can install them, and
what changes once a cube lives in `node_modules` instead of in your own tree.
If you only want to *use* a published bundle, you need one line of config:
```json
{ "cubePackages": ["@bitsquare/cubes-core"] }
```
The rest of this document is for writing one.
- [What a bundle is](#what-a-bundle-is)
- [The package manifest](#the-package-manifest)
- [Writing the cubes](#writing-the-cubes)
- [Ids are claimed globally](#ids-are-claimed-globally)
- [An installed bundle is read-only](#an-installed-bundle-is-read-only)
- [How resolution actually works](#how-resolution-actually-works)
- [Publishing](#publishing)
- [Troubleshooting](#troubleshooting)
## What a bundle is
An ordinary npm package that ships cube directories and points at them from its
own `package.json`. There is no build step, no plugin API and no entry point —
nopy reads the directories off disk and imports each `manifest.mjs` directly.
```
@acme/cubes-web
├── package.json nopy.cubes → ["./cubes"]
├── README.md
└── cubes/
├── nginx/
│ ├── manifest.mjs
│ └── deploy.py
└── certbot/
├── manifest.mjs
└── deploy.py
```
`@bitsquare/cubes-core` in this repository is the worked example, and is consumed
by this repository through exactly the mechanism described here — it is not
special-cased.
## The package manifest
```json
{
"name": "@acme/cubes-web",
"version": "1.0.0",
"type": "module",
"nopy": { "cubes": ["./cubes"] },
"files": ["cubes", "!cubes/**/*.log", "README.md", "LICENSE"],
"publishConfig": { "access": "public" },
"dependencies": {
"@bitsquare/nopy-cube": "^1.0.0",
"zod": "^4.4.3"
}
}
```
**`nopy.cubes`** is the only field nopy requires. It is an array of directories,
relative to the package root, each scanned recursively for cubes. Several
entries are fine; a single `["./cubes"]` is the norm. Every entry must exist and
must stay inside the package — a path escaping the root is refused, not resolved.
**`type: "module"`** matters: manifests are ESM. Without it a `manifest.mjs` still
loads (the extension carries the day), but anything it imports relatively will
not behave the way you expect.
**`files`** decides the tarball. Note the negation: a cube that has been run
leaves a `pyinfra-debug.log` next to its `deploy.py`, and `.gitignore` has no
effect on what npm packs. Check with `npm pack --dry-run` before publishing.
**Dependencies** are `@bitsquare/nopy-cube` and `zod`, both real dependencies
rather than peers — a bundle is a leaf, and the copies it gets are the copies its
manifests use. Do **not** depend on `@bitsquare/nopy`: the CLI is what installs
your bundle, not the other way round, and depending on it invites two copies of
the same code into one tree.
## Writing the cubes
A cube directory holds a manifest (`manifest.mjs` or `*.manifest.mjs`) and a
deploy script (`deploy.py` or `*.deploy.py`). Anything else in the directory is
invisible to the loader but readable from the script, which runs with the cube
directory as its working directory.
```javascript
// cubes/nginx/manifest.mjs
import { Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
export default Manifest({
id: 'web:nginx',
name: 'Install and configure nginx',
dependencies: () => ['apt:essentials'],
secrets: ['TLS_KEY'],
schema: z.object({
SERVER_NAME: z.string().describe('Server name').default('example.com'),
TLS_KEY: z.string().describe('TLS private key (PEM)').default(''),
HTTP2: z.boolean().describe('Enable HTTP/2').default(true),
}),
});
```
```python
# cubes/nginx/deploy.py
from pyinfra import host
from pyinfra.operations import apt, files
SERVER_NAME = host.data.SERVER_NAME
apt.packages(name='Install nginx', packages=['nginx'], _sudo=True)
```
Import **`@bitsquare/nopy-cube`**, not `@bitsquare/nopy`. It is types and a
factory with zod as its only peer — no CLI, no prompts, no process spawning — so
your bundle stays a leaf. (`@bitsquare/nopy` re-exports the same surface as
`cubes.Manifest`, which is what older manifests use. It still works; it just
drags the CLI into your dependency graph if you declare it.)
Four things the schema is load-bearing for:
- **`.describe()` is the prompt label.** A field without one prompts with its raw
key.
- **`.default()` makes the field optional.** A field with no default is required,
and is re-prompted on replay if a session has no value for it.
- **Every schema key reaches pyinfra** as `--data KEY=value`, so `host.data.KEY`
is always defined. pyinfra parses the values itself: `"true"` arrives as a
bool, `"8080"` as an int.
- **`secrets` names keys whose values must not be persisted.** They are excluded
from session files and history, masked wherever a command is printed, and
re-prompted on replay. Naming a key that is not in the schema is a load error.
A secret is still visible in `ps` while pyinfra runs — masking covers nopy's
own output, not the process table — so treat it as protection against writing
credentials to disk, not as protection against a shared host.
`dependencies` is a function of the *collected* variables, so it can branch on
what the user actually answered, and it may pass parameters:
```javascript
dependencies: (v) => (v.HTTP2 ? ['apt:essentials', ['web:tls', { MODE: 'strict' }]] : []),
```
`before` / `after` hooks get a context whose `exec(id, vars)` pulls in any cube
by id, declared dependency or not. See [HOOKS.md](HOOKS.md).
## Ids are claimed globally
An id is claimed across every source at once — `cubeDirs`, `.npcubes` trees and
every installed bundle share one flat namespace. Two cubes claiming the same id
abort the run with an error naming both and where each came from.
There is no precedence and no shadowing, deliberately, in either direction: a
local cube does not quietly win over a packaged one, and installing a second
bundle cannot silently change what an existing id deploys. Overriding a cube from
a bundle is not a supported operation; fork the cube under your own id instead.
So prefix distinctly. `@acme/cubes-web` claiming `nginx` is asking for trouble the
first time someone installs a second bundle; `web:nginx` is not. Ids need not
mirror the directory layout — `cubes/network/tailscale` declares `net:tailscale`
— so the prefix is free.
An id is also the session key. Renaming one silently invalidates every recorded
session that used it, so treat a rename as a breaking change of the bundle.
## An installed bundle is read-only
Under pnpm, installed files are **hardlinked into a global store shared by every
project on the machine**. A cube that writes next to its own `deploy.py` does not
just dirty one `node_modules` — it corrupts that store for every other project.
Write to `/tmp`, to a path the user configured, or to the remote host. Never to
the cube's own directory. Files the cube needs to *read* (templates, config
fragments, systemd units) are fine and are exactly what the cube directory is for
— `deploy.py` runs with it as the working directory, so `files.template('nginx.conf.j2', ...)`
resolves.
This is the one constraint that does not exist while the cubes live in your own
repo, which makes it the one most likely to be discovered late. Test against an
installed copy, not a linked one.
## How resolution actually works
Worth knowing, because two of the failure modes are otherwise baffling.
**Where a package is looked up from.** Each `cubePackages` entry is resolved from
the directory of the config file that named it, not from the working directory.
Configs merge upward, so a `.nopyrc.json` two levels up can name a bundle that
only exists in *its* `node_modules`, and it resolves. The lookup reads
`package.json` off disk via `createRequire(...).resolve.paths()` rather than
going through `exports` — a bundle ships directories and has no entry point to
declare.
**Why the loader does not simply scan `node_modules`.** It cannot: pnpm plants a
symlink at `node_modules/<name>`, and `readdir` reports it as a symlink, not a
directory, so a recursive scan skips every package silently. Naming packages
explicitly is the fix, and it is also the reason `node_modules` is skipped during
the cube scan itself.
**How a manifest finds its imports.** Ordinary Node resolution, from the
manifest's own directory. An installed bundle has its own `node_modules` with
`@bitsquare/nopy-cube` and `zod` in it, so this just works. A hand-written cube
sitting in a directory with no `node_modules` would historically fail with
`ERR_MODULE_NOT_FOUND`; nopy now registers a resolve hook that catches exactly
that case and falls back to resolving `@bitsquare/nopy-cube`, `@bitsquare/nopy`
and `zod` from the running CLI. Normal resolution is always tried first, so a
cube that ships its own zod keeps it. Treat the hook as a convenience for local
cubes — a published bundle must declare its dependencies properly.
**Two copies of zod is a real hazard.** `instanceof` comparisons fail across
copies, which is why nopy inspects schemas structurally (`schema.def.type`) and
why `secrets` is a plain array rather than `.meta()` metadata — zod's metadata
registry is per-copy, and a marker written into one copy's registry is invisible
to another. Keep your zod range compatible with the CLI's (`^4.4.3`) and the
package manager will usually give you one copy.
## Publishing
Nothing bundle-specific: `npm publish` (or `pnpm publish`) with a version bump.
Some things worth deciding once:
- **Version the bundle independently of nopy.** There is no compatibility check
between the two — the loader reads whatever `nopy.cubes` points at. Document
the nopy version you test against in your README.
- **Renaming or removing an id is breaking.** It invalidates recorded sessions
and breaks any manifest listing it as a dependency, including manifests in
other people's bundles.
- **Changing a schema key is breaking** in the same way; adding one with a
`.default()` is not.
- **Test the installed shape, not the linked one.** `npm pack`, install the
tarball into a throwaway directory with a `.nopyrc.json` naming it, and deploy
from it. This is what catches a missing file, a cube that writes to its own
directory, and an undeclared dependency — none of which show up while the
package is symlinked into the repo that wrote it.
For an unattended check, replay a session file rather than reaching for `-P`
alone, which still opens the interactive picker:
```sh
nopy install -l session.json -P -D
```
Note that a replay re-prompts for anything a manifest lists in `secrets` —
those are never written to a session — so pick a cube without them, or put the
values under `env` in `.nopyrc.json`.
For how this repository releases its own packages, see
[README.PUBLISH.md](../../../README.PUBLISH.md).
## Troubleshooting
| Symptom | Cause |
| --- | --- |
| `Cube package 'X' is not installed (looked up from …)` | Not installed, or installed somewhere other than the config that named it. The path in the message is where the lookup started. |
| `Cube package 'X' declares no cubes` | Missing or malformed `nopy.cubes` in the package's `package.json`. It must be a non-empty array of strings. |
| `'./cubes' does not exist in …` | The directory was not packed. Check `files` and `npm pack --dry-run`. |
| `'…' points outside the package` | A `nopy.cubes` entry escaping the package root. Not allowed. |
| `Duplicate cube id 'X' from N sources:` | Two or more cubes claiming one id; the message lists each source. Rename one — there is no precedence rule to lean on. |
| `ERR_MODULE_NOT_FOUND` for `zod` or `@bitsquare/nopy-cube` | The bundle did not declare them as dependencies. The resolve-hook fallback covers loose local cubes, not published packages. |
| `Invalid manifest in …: 'secrets' names X, which is not in the schema` | A `secrets` entry with no matching schema key — usually a typo or a renamed field. |
| Cubes work linked, fail installed | Almost always a write into the cube's own directory, or a file missing from `files`. |
packages/nopy/docs/CUBE-PACKAGES.md
+154 -45
View File
@@ -1,6 +1,11 @@
# Cube bundles as npm packages
Status: **Phase 0 has landed; Phases 1–6 are still a plan, not a record.**
Status: **All six phases have landed. This document is now a record, not a plan.**
The one thing still unproven is the publish lane against a real registry — see
*Risks*.
`cubePackages` resolves and loads end to end, `@bitsquare/nopy-cube` exists and
the publish lane can ship a linked package. What is missing is a bundle to point
`cubePackages` at.
Distributing cubes as npm packages so a project can `pnpm add @acme/cubes-net`
and have its cubes show up in `nopy` alongside local ones.
@@ -166,7 +171,7 @@ Rules:
- A bundle must not ship a `.nopyrc.json`. Config discovery walks up from
`process.cwd()`, never from cube directories, so it would never be read.
## Phase 2 — resolution
## Phase 2 — resolution — **done**
### Config surface
@@ -271,7 +276,7 @@ it is exported from `src/cubes/index.ts` and covered by tests. The
`node_modules` skip inside `scanDirectory` stays and is now *correct*: a
bundle's own `node_modules` should not be scanned.
## Phase 3 — hard errors with attribution
## Phase 3 — hard errors with attribution — **done**
`Cube` gains a source, as an optional fourth constructor parameter so the public
signature stays backwards compatible:
@@ -306,7 +311,7 @@ claim the same id they are mutually exclusive, and the fix is upstream.
Surface the source in the interactive picker and in `--json` output so a user can
see where a cube came from before running it.
## Phase 4 — `@bitsquare/nopy-cube`, the authoring package
## Phase 4 — `@bitsquare/nopy-cube`, the authoring package — **done**
The problem: a manifest does `import { cubes } from '@bitsquare/nopy'`, resolved
by ordinary Node resolution from the manifest's own directory. From inside
@@ -350,20 +355,33 @@ already coverage-excluded barrels — so `import { cubes } from '@bitsquare/nopy
in every existing manifest keeps working unchanged. Nothing in `cubes/` has to be
touched at migration time.
Repo plumbing this requires:
`cubes/types.ts` and `cubes/factories.ts` moved wholesale, with
`tests/cubes.types.test.ts` and `tests/cubes.factories.test.ts` behind them.
`tests/helpers/foreign-zod.ts` is duplicated rather than shared — fifteen lines,
and the alternative is a test-only dependency edge between the packages.
- `tsconfig.base.json`: add `"@bitsquare/nopy-cube": ["./packages/nopy-cube/src"]`
to `paths`.
- Root `tsconfig.json`: add the project reference.
- `packages/nopy/tsconfig.json`: `references` is currently `[]` — add
`{ "path": "../nopy-cube" }`. This is the first reference edge in the repo, so
`tsc --build` ordering starts mattering.
Repo plumbing it took:
- `tsconfig.base.json`: `"@bitsquare/nopy-cube": ["./packages/nopy-cube/src"]`.
- Root `tsconfig.json` and `packages/nopy/tsconfig.json`: the project reference.
This is the first reference edge in the repo, and it broke the gate
immediately: **`tsc --build --noEmit` is not legal once a project has
references** — TS6310, "referenced project may not disable emit", because a
composite project has to emit the declarations its dependents read. The root
`typecheck` script is now plain `tsc --build`. It still fails on a type error,
and it now also proves the build works; the cost is that it writes `dist`,
which is gitignored.
- `packages/nopy/package.json`: `"@bitsquare/nopy-cube": "workspace:*"`.
- A `vitest.config.ts` for the new package with the same thresholds. `Manifest()`,
`Manifest.create()` and `Cube.getDefaults()` all carry logic, so the relevant
cases move over from `tests/cubes.factories.test.ts`.
- `packages/nopy/vitest.config.ts`: a `resolve.alias` for `@bitsquare/nopy-cube`
pointing at `../nopy-cube/src/index.ts`. Without it the workspace link
resolves through `exports` to `dist`, so `pnpm test` on a clean checkout would
fail until something had built it, and a stale `dist` would silently be what
the tests ran against. The same config excludes `**/nopy-cube/**` from
coverage — the aliased files were being counted against nopy's thresholds.
- A `vitest.config.ts` for the new package with the same thresholds. It sits at
100 % statements/functions/lines, 91 % branches.
### The release lane needs fixing first
### The release lane needed fixing first
This is the part that is easy to miss. `link-workspace-packages` is unset and
pnpm 10+ defaults it to `false`, so a plain semver range would resolve
@@ -389,10 +407,40 @@ Pick one before publishing anything:
passes — compute every snapshot version first, then publish — so `nopy` can pin
the exact `nopy-cube` snapshot from the same run.
Recommendation: `pnpm publish`, and verify against the Gitea registry with a
**Measured, both directions.** `npm pack` in `packages/nopy` produces a tarball
whose manifest still reads `"@bitsquare/nopy-cube": "workspace:*"`; `pnpm pack`
produces one that reads `"1.0.0-alpha0"`. So the failure was real and the fix
works.
Went with `pnpm publish --ignore-scripts --no-git-checks` in both workflows.
`--no-git-checks` is not optional in either: `release.yml` runs on a detached
HEAD, and `publish-snapshot.yml` dirties the tree by stamping versions.
(`pnpm pack` has no `--ignore-scripts`, only `pnpm publish` does.)
Three small scripts carry the parts that are easy to get wrong, all runnable
locally:
- **`scripts/verify-pack.mjs`** — packs every publishable package and fails if a
`workspace:` range survived into the tarball. Runs between build and publish
in both workflows. Turns "npm would have shipped a broken manifest" from an
install-time surprise into a red run.
- **`scripts/publish-order.mjs`** — topologically sorts the publishable packages.
`packages/*/` alphabetically puts `nopy` ahead of the `nopy-cube` it depends
on; the snapshot workflow now iterates this instead.
- **`scripts/linked-deps.mjs`** — lists a package's workspace links as
`<name> <version>`, resolved by package name rather than by directory.
`release.yml` uses it to refuse a release whose linked dependency is not on
npmjs yet, which is the one mistake that cannot be taken back after 72 hours.
`publish-snapshot.yml` also became two passes over the packages: stamp every
version first, then publish. `pnpm publish` substitutes the version the linked
package declares *at pack time*, so `nopy-cube` has to be carrying its snapshot
version before `nopy` is packed.
Still unverified: none of this has run against the Gitea registry. Worth a
throwaway version before the first real release.
### Also: the resolve hook
### Also: the resolve hook — built
Independent of the split, and worth building anyway — it retires the
`ERR_MODULE_NOT_FOUND` gotcha CLAUDE.md documents for the local `cubes/` tree,
@@ -414,33 +462,48 @@ Falling back for `zod` hands local cubes the *CLI's* zod instance, so no
duplication arises there. Bundles are the case that duplicates it, and Phase 0.4
is what makes that safe.
New `packages/nopy/src/nopy.resolve-hook.mjs`, registered once from `loadCubes()`
`packages/nopy/src/cubes/resolve-hook.mjs`, registered once from `loadCubes()`
before the first `import(manifestPath)`:
```ts
module.register('./nopy.resolve-hook.mjs', import.meta.url, {
data: { fallback: import.meta.resolve('./index.js') },
});
module.register('./resolve-hook.mjs', import.meta.url, { data: { from: import.meta.url } });
```
The hook tries `next(specifier, ctx)` **first** and only falls back to the
running CLI's own copy on failure. That ordering matters: a consumer that has its
own `@bitsquare/nopy` installed keeps using it, so the hook never silently
introduces version skew.
`from` is a URL inside the running CLI's own package; the hook thread builds a
`createRequire` from it and resolves the fallbacks out of the CLI's own
dependencies.
Constraints:
The hook tries `next(specifier, ctx)` **first** and only falls back on failure.
That ordering matters: a consumer that has its own copy installed keeps using
it, so the hook never silently introduces version skew. There is a test for
exactly that — a stub `zod` beside the cube wins over the CLI's real one.
- `module.register()` is process-global and cannot be undone. Install it once,
behind a module-level guard.
Constraints, as built:
- `module.register()` is process-global and cannot be undone. Installed once,
behind a module-level guard, and wrapped in a `try` — the hook is a
convenience, so a registration failure must not abort a run.
- The hook file runs on a separate thread; the `data` payload must be
structured-cloneable (a string URL is).
- The `.mjs` must ship in `dist` and be listed in `files` — it already is, via
the `dist` entry.
- It resolves `@bitsquare/nopy` and `zod`, not `@bitsquare/nopy-cube`. Bundles
never depend on the hook; only the in-repo `cubes/` tree and hand-written local
cubes do.
- The `.mjs` has to reach `dist`, and `tsc` does not copy it: nopy's `build` is
now `tsc && cp src/cubes/*.mjs dist/cubes/`. `files` already covers it via the
`dist` entry.
- It covers **three** specifiers, not the two the plan named: `zod`,
`@bitsquare/nopy`, and `@bitsquare/nopy-cube` — a hand-written local cube is
as entitled to the new authoring package as to the old one. Subpaths count
(`@bitsquare/nopy/package.json`), anything else stays a hard failure.
## Phase 5 — proof of concept: `packages/cubes-core`
**The tests have to spawn a real `node`.** Written inside the vitest worker they
pass whether or not the hook is installed: vite resolves the dynamic import
itself and finds `zod` from the project root. `tests/cubes.resolve-hook.test.ts`
therefore runs each case in a child process, and the first case asserts the
*failure* without the hook so the rest cannot silently stop proving anything.
**Verified end to end.** From a plain `node` at the repo root, with nothing
linked, the built loader reads all 22 cubes under `cubes/` with zero errors. The
`ERR_MODULE_NOT_FOUND` gotcha in `CLAUDE.md` is retired.
## Phase 5 — proof of concept: `packages/cubes-core` — **done**
Depends on Phase 4 shipping first — the bundle cannot declare
`@bitsquare/nopy-cube` as a dependency until it exists, and the publish-lane fix
@@ -478,19 +541,50 @@ has to be in place before either package is published.
`nopy-cube` references from Phase 4 are separate.)
8. Biome already lints `cubes/**/*.mjs` from the root; only the path changes.
### Verifying the PoC
### What differed from the plan
- **In-workspace:** `pnpm --filter @bitsquare/nopy run nopy -P` from the repo
root lists `net:tailscale`, `apt:install`, … and prints deploy commands whose
`--chdir` points into `node_modules/@bitsquare/cubes-core/cubes/…`.
- **Out-of-workspace (the real test):** `npm pack` the bundle, install the
tarball into a throwaway directory with a `.nopyrc.json` naming it, install
`nopy` *globally*, and run `nopy -P`. This is what actually exercises Phase 4 —
a manifest resolving its import from a `node_modules` tree that has no
`@bitsquare/nopy` in it. Check the installed tarball's `package.json` really
carries a concrete `@bitsquare/nopy-cube` range and not `workspace:*`.
- **Step 2's optional migration was done.** All 22 manifests now import
`{ Manifest }` from `@bitsquare/nopy-cube`, not `{ cubes }` from
`@bitsquare/nopy`. Optional for correctness, but it is the only version of the
PoC that proves anything: leaving the old import in place would have resolved
through the CLI that happens to sit in the same tree.
- **`uniqid` had to move too.** Two manifests use it (`admin:hostname` bare,
`user:add` via `cubes.uniqid`), so `src/cubes/utils.ts` and its test went to
`nopy-cube` alongside `types.ts`, and `uniqid` joined the authoring barrel.
Otherwise one migrated manifest would still have been importing the CLI.
- **`files` needs a log exclusion.** Cubes that have been run leave a gitignored
`pyinfra-debug.log` next to `deploy.py`; gitignore does not filter an npm
tarball. `"files": ["cubes", "!cubes/**/*.log", …]` does. Verified: 22
manifests, 22 deploy scripts, 0 logs in the packed artefact.
- **`verify-pack.mjs` picks the bundle up for free** — it walks every non-private
`packages/*`, so `cubes-core`'s `workspace:*` edge is checked like nopy's.
## Phase 6 — documentation
### Verifying the PoC — done
- **In-workspace:** the built loader, run from the repo root against the new
root `.nopyrc.json`, reads 22 cubes with 0 errors and reports
`source: { type: 'package', packageName: '@bitsquare/cubes-core', dir:
'…/node_modules/@bitsquare/cubes-core/cubes' }` — the pnpm symlink path, not a
plain directory.
- **Out-of-workspace (the real test):** `pnpm pack` for `nopy-cube`, `nopy` and
`cubes-core`, then **`npm install`** of all three tarballs into a throwaway
directory with a `.nopyrc.json` naming only the bundle. npm is the strict test
here — it does not understand `workspace:`, so a leaked range fails the install
outright. It installed clean, and the installed
`@bitsquare/nopy/package.json` carries `"@bitsquare/nopy-cube":
"1.0.0-alpha0"`. `nopy install -l session.json -P -D` then resolved
`apt:essentials` and printed a `--chdir` into
`node_modules/@bitsquare/cubes-core/cubes/apt/essentials`. Since the loader
aborts on any manifest error and this run did not, all 22 manifests imported
`@bitsquare/nopy-cube` and `zod` successfully from a tree containing no
workspace links.
Note for anyone repeating this: `-P` on its own is interactive, and a replay
still prompts for anything a manifest declares in `secrets` (they are never
persisted to a session) — `net:tailscale` will sit there waiting. Use a
session file with a cube that has no secrets, or answer the prompt.
## Phase 6 — documentation — **done**
- `CLAUDE.md`: the repo table gains two rows (`packages/nopy-cube`,
`packages/cubes-core`) and loses the `cubes/` one; "The two packages do not
@@ -503,6 +597,21 @@ has to be in place before either package is published.
plus the ordering constraint — `nopy-cube` releases before anything that
depends on it.
Beyond the list: `CLAUDE.md` also needed the `typecheck` command corrected
(`tsc --build`, not `--noEmit` — see Phase 4), a note on the vitest source alias
and the coverage exclusion, and three entries under *Known drift*. `README.PUBLISH.md`
absorbed the whole publish-lane rework, not just the tag prefixes: `pnpm publish`
over `npm publish` and why, the two-pass version stamping, `verify-pack.mjs`,
`publish-order.mjs`, `linked-deps.mjs`, and a local rehearsal recipe that uses
**npm** to install the tarballs precisely because npm is the one that rejects a
leaked `workspace:` range.
One workflow change came out of writing this up: `ci.yml` now runs
`verify-pack.mjs` too. It was only in the two publish workflows, which means a
leaked range would have failed the release rather than the pull request that
introduced it — the wrong end of the process for a mistake that is free to catch
early.
## Testing
The coverage gate (85 % branches/functions, 80 % lines/statements, per package)
packages/nopy/docs/REFACTORING.md
+26
View File
@@ -54,3 +54,29 @@ This document tracks the major refactoring of the `nopy` package.
- `VariableAssignment` offers every schema key, not only the ones carrying a default, and shows the value the run would actually use as the initial.
- **Proposed Solution**: (Done)
### 6. Make variable assignment a first-class concept
- **Status**: ✅ Completed
- **Goal**: Give a variable an identity and a provenance, instead of inferring both from which bag it happened to sit in.
- **Rationale**: Item 5 left precedence encoded as the field order of an object literal inside `Variables.get()` — `defaults`, then `global`, then `prompts`, then `params`. Nothing named the ranking, nothing could be asked where a value came from, and a replay had to be smuggled into the `prompts` bag because there was no origin that meant "recorded". Every question that followed — what should a session record, which values are safe to print — needed provenance to answer.
- **Context**:
- `Assignment { value, origin }` and an `Origin` ranked `default(0) < env(1) < session(2) < prompt(3) < param(4)`. Precedence is now data, not the order lines appear in.
- `Variable` is a class over an assignment list. `assignments` is the true history, newest first and never reordered; `ordered` is a *stable* sort of it by origin rank, and `value`/`origin` read the head of that. Stability is what makes the two views coexist: same-origin ties keep the newest in front while the value it displaced stays visible.
- The `global` bag is gone. Config `env` is seeded per cube as a real assignment at origin `env`, so `variables.get('global')` — a cube id that was never a cube — is no longer a thing.
- Replay assigns at origin `session`, which outranks `env` and `default` on its own. The `prompts`-bag workaround is deleted.
- A session records `Variables.persistable()` — every effective value, not just prompted ones. A `-D` run used to record nothing and replay by re-deriving from whatever the defaults said at replay time.
- **Trade-off accepted**: recorded values now outrank the current `.nopyrc.json` `env` and the current schema defaults, so editing either no longer leaks into an existing session's replay. That is the point of a snapshot, but it does mean picking up a new default requires re-recording.
- **Proposed Solution**: (Done)
### 7. Manifest-declared secrets
- **Status**: ✅ Completed
- **Goal**: Let a manifest say which schema keys hold sensitive values, and act on it.
- **Rationale**: Item 6 made sessions record everything, which forced the question of what must *not* be recorded. The codebase already had an answer of sorts — `outputExecutionPlan` masked any variable whose name contained "password" — that missed `TOKEN`, `PSK` and `AUTH_KEY`, and was defeated anyway by the unmasked command printed one line above it.
- **Context**:
- `Manifest.secrets?: string[]`, validated at load: an entry that is not a key of `schema` is a manifest error and aborts the run, so a typo cannot silently leave a value unprotected.
- Deliberately a plain array, not zod metadata. `.meta()` and `.describe()` store into `z.globalRegistry`, which is per-copy — a manifest built by a different zod copy would look up empty. Fail-open is fine for a missing prompt label and unacceptable for a secret marker.
- `maskCommand()` replaces declared `--data` values and the SSH `--password` in the command string itself, and is wired into `--print-only`, the dry-run plan and the debug log. The `nopy` logger runs at `lowestLevel: 'debug'`, so that last one was printing credentials on every run.
- Secrets are excluded from `persistable()`, so a replay has a gap where one used to be. `fillSessionGaps` prompts for `requiredKeys() ∪ secrets`; under `-D` it fails naming them, consistent with item 5's fail-fast.
- **Scope limit**: `secrets` keeps a value out of what nopy writes. The value is still on pyinfra's command line (visible in `ps`), still echoed by the variable form, and a `.default()` is still plain text in the manifest. Documented rather than fixed — the first is inherent to pyinfra's interface.
- **Bug fixed along the way**: `cubes/user/add` generated a random password as its schema `.default()`. Because the key had a default it was never in `requiredKeys()`, and because a generated default is re-evaluated on every read, an unattended run created an account with a credential nobody had seen and a replay created a different one again. It is now the literal `changeme`.
- **Proposed Solution**: (Done)
packages/nopy/docs/SESSION_FORMAT.md
+15 -1
View File
@@ -294,8 +294,9 @@ export default {
2. **Document your cubes** - Add comments explaining what each cube does
3. **Use environment variables** - Make sessions reusable across environments
4. **Extract common config** - Share configuration across multiple sessions
5. **Version control** - Both formats work well with git
5. **Version control** - Both formats work well with git, but a recorded session holds every value its run used; read one before committing it
6. **Validate at runtime** - The loader validates the structure regardless of format
7. **Leave secrets out** - Declare them in the manifest instead, and let the replay ask
## Session Schema
@@ -321,3 +322,16 @@ interface AuthSession {
username?: string;
}
```
A session nopy *writes* holds, per cube, every value that cube ran with — what
was typed, what came from `.nopyrc.json`, what a dependency supplied, and what
fell through to the schema's `.default()`. Two things are deliberately absent and
are asked for again on replay: the SSH password, and any key the cube's manifest
listed under `secrets`.
A session you write by hand is under no such obligation — `variables` may hold as
few keys as you like, and anything missing resolves the usual way. Note that a
key declared a secret is prompted for whether or not the session carries a value:
writing one in only pre-fills the prompt, it does not skip it. The variable form
shows what it is editing, so a secret you put in a session file appears on screen
as well as on disk.
packages/nopy/package.json
+2 -1
View File
@@ -43,7 +43,7 @@
},
"scripts": {
"clean": "rm -rf dist .tsbuildinfo",
"build": "tsc",
"build": "tsc && cp src/cubes/*.mjs dist/cubes/",
"prepack": "pnpm run build",
"link:local": "pnpm run build && npm link",
"nopy": "tsx src/nopy.cli.ts",
@@ -54,6 +54,7 @@
"test:watch": "vitest"
},
"dependencies": {
"@bitsquare/nopy-cube": "workspace:*",
"@logtape/logtape": "^2.2.4",
"commander": "^15.0.0",
"enquirer": "^2.4.1",
packages/nopy/src/cubes/dependencies.ts
+59 -10
View File
@@ -3,13 +3,13 @@
* @module cubes/dependencies
*/
import type { Cube, CubeVariables, HookContext } from '@bitsquare/nopy-cube';
import { getLogger } from '@logtape/logtape';
import type { Variables } from '../nopy.common.js';
import type { NopyConfig } from '../nopy.config.js';
import type { DeployCall } from '../nopy.executor.js';
import { VariableAssignment } from '../nopy.prompts.js';
import type { CubeSession, NopySession } from '../nopy.session.js';
import type { Cube, CubeVariables, HookContext } from './types.js';
const log = getLogger(['nopy', 'resolution']);
@@ -38,6 +38,12 @@ export class BuildContext {
} = {}
) {}
/** Required schema keys that nothing has supplied a value for. */
private missingRequired(cube: Cube): string[] {
const resolved = this.variables.get(cube.id);
return cube.requiredKeys().filter((key) => resolved[key] === undefined);
}
/**
* Fails a non-interactive run that cannot fill a required variable.
*
@@ -45,8 +51,7 @@ export class BuildContext {
* `--data`, and the deploy script would read `None` off `host.data`.
*/
private assertVariablesComplete(cube: Cube): void {
const resolved = this.variables.get(cube.id);
const missing = cube.requiredKeys().filter((key) => resolved[key] === undefined);
const missing = this.missingRequired(cube);
if (missing.length === 0) return;
const [one, them] =
@@ -58,6 +63,43 @@ export class BuildContext {
);
}
/**
* Asks for the variables a replay cannot supply on its own.
*
* Two kinds. Required keys can be absent because the session predates them or
* was recorded by a `--use-defaults` run. Secrets are absent by design: they
* are never written to a session, so replaying without asking would deploy a
* cube with the key missing — or, for a secret carrying a default, with a
* value silently different from the run being replayed.
*
* Secrets are asked for even when a default did fill them in, which is why
* this cannot key off "has no value": the whole point is that the recorded
* answer is gone and only the user knows what it was.
*/
private async fillSessionGaps(cube: Cube): Promise<void> {
const gaps = [...new Set([...this.missingRequired(cube), ...cube.secrets])];
if (gaps.length === 0) return;
if (this.options.useDefaults) {
throw new Error(
`Cube "${cube.id}" cannot be replayed with --use-defaults: ${gaps.join(', ')} ` +
'would have to be entered. Secrets are never recorded in a session. ' +
'Replay without --use-defaults, or set the values under "env" in .nopyrc.json.'
);
}
log.debug('Filling session gaps', { cubeId: cube.id, gaps });
await VariableAssignment(cube, this.variables, { keys: gaps });
// A cancelled form leaves the run short of a value it cannot invent.
const stillMissing = this.missingRequired(cube);
if (stillMissing.length > 0) {
throw new Error(
`Cube "${cube.id}" is missing ${stillMissing.join(', ')} and cannot be deployed.`
);
}
}
/**
* Resolves a cube, its dependencies, and hooks recursively
*/
@@ -73,20 +115,22 @@ export class BuildContext {
log.debug('Resolving cube', { cubeId, host });
// 1. Assign overrides and defaults
// 1. Declare secrets, then assign overrides and defaults. Declaring first
// means even the config `env` seeded on the cube's first assignment is
// already marked, so nothing reaches a session or a log unredacted.
this.variables.declareSecrets(cubeId, cube.secrets);
if (Object.keys(overrides).length > 0) {
this.variables.assign(cubeId, 'params', overrides);
this.variables.assign(cubeId, 'param', overrides);
}
this.variables.assign(cubeId, 'defaults', cube.getDefaults());
this.variables.assign(cubeId, 'default', cube.getDefaults());
// 2. Variable collection
if (this.options.isSessionReplay) {
// Recorded answers go back into the scope they came from, so a replay
// reproduces them even when `env` sets the same key to something else.
const sessionCube = this.session.cubes.find((c) => c.key === cubeId);
if (sessionCube) {
this.variables.assign(cubeId, 'prompts', sessionCube.variables);
this.variables.assign(cubeId, 'session', sessionCube.variables);
}
await this.fillSessionGaps(cube);
} else if (this.options.useDefaults) {
log.debug('Skipping prompts, using defaults', { cubeId });
this.assertVariablesComplete(cube);
@@ -155,13 +199,18 @@ export class BuildContext {
cwd: cube.dir,
command,
env: cubeVars,
secrets: cube.secrets,
dependencies: [],
});
if (!this.cubeSessions.some((s) => s.key === cubeId)) {
// Every value the run settled on, not just the prompted ones — otherwise a
// `--use-defaults` run records nothing and replaying it re-derives from
// whatever the defaults and `env` happen to say now. Secrets are the one
// exclusion; a replay asks for those again.
this.cubeSessions.push({
key: cubeId,
variables: this.variables.get(cubeId, 'prompts'),
variables: this.variables.persistable(cubeId),
});
}
packages/nopy/src/cubes/index.ts
+22 -19
View File
@@ -6,34 +6,37 @@
* @module cubes
*/
// Dependencies
export { BuildContext } from './dependencies.js';
// Factory functions
export {
createManifest,
manifest,
} from './factories.js';
// Loader
export {
findCubeDirectories,
getCube,
loadCubes,
} from './loader.js';
// The authoring surface lives in its own package so that a cube bundle can
// depend on it without pulling the CLI in. Re-exported here so that
// `import { cubes } from '@bitsquare/nopy'` in a manifest keeps working.
export type {
AnyObjectSchema,
CubeSource,
CubeVariables,
DependencySpec,
Hook,
HookContext,
LoadResult,
} from './types.js';
// Types
} from '@bitsquare/nopy-cube';
export {
Cube,
createManifest,
Manifest,
manifest,
uniqid,
zodInner,
zodKind,
} from './types.js';
// Utilities
export { uniqid } from './utils.js';
} from '@bitsquare/nopy-cube';
// Dependencies
export { BuildContext } from './dependencies.js';
// Loader
export type { CubeRoot } from './loader.js';
export {
findCubeDirectories,
findCubeRoots,
getCube,
loadCubes,
} from './loader.js';
// Packages
export type { CubePackage } from './packages.js';
export { resolveCubePackages } from './packages.js';
packages/nopy/src/cubes/loader.ts
+102 -17
View File
@@ -3,21 +3,55 @@
* @module cubes/loader
*/
import module from 'node:module';
import path from 'node:path';
import { Cube, type CubeSource, type LoadResult, type Manifest } from '@bitsquare/nopy-cube';
import { z } from 'zod';
import { fs } from 'zx';
import { loadConfig } from '../nopy.config.js';
import { Cube, type LoadResult, type Manifest } from './types.js';
import { resolveCubePackages } from './packages.js';
let hookRegistered = false;
/**
* Traverses upwards from the current working directory to the root
* and collects all directories that contain a `.npcubes` marker file.
* Installs the fallback resolver that lets a manifest in a bare directory
* import `@bitsquare/nopy-cube` or `zod` — see `resolve-hook.mjs`.
*
* Also includes directories specified in the `.nopyrc.json` configuration.
*
* @returns Array of absolute paths to directories containing cubes
* `module.register()` is process-global and cannot be undone, so this runs once
* and only when cubes are about to be imported. Registration failing is not
* worth aborting a run over: without the hook, a cube that needed it fails on
* its own import with a message that names the file.
*/
export function findCubeDirectories(): string[] {
function registerResolveHook(): void {
if (hookRegistered) return;
hookRegistered = true;
try {
// `from` is a URL inside this package, so the hook thread resolves the
// fallbacks out of the running CLI's own dependencies.
module.register('./resolve-hook.mjs', import.meta.url, { data: { from: import.meta.url } });
} catch {
// Nothing to do: the hook is a convenience, never load-bearing.
}
}
/** A directory to scan, and what put it in the list. */
export interface CubeRoot {
dir: string;
source: CubeSource;
}
/**
* Collects every root to scan for cubes:
*
* - `cubeDirs` from the merged configuration,
* - every ancestor of the working directory holding a `.npcubes` marker file,
* - the cube directories of every package named in `cubePackages`.
*
* Only the last of those can fail — a missing directory is ignored, a missing
* package is not (see `resolveCubePackages`).
*/
export function findCubeRoots(): { roots: CubeRoot[]; errors: string[] } {
let currentDir = process.cwd();
const config = loadConfig();
const dirSet = new Set<string>(config.cubeDirs.map((dir) => path.resolve(process.cwd(), dir)));
@@ -38,7 +72,29 @@ export function findCubeDirectories(): string[] {
currentDir = parentDir;
}
return [...dirSet];
const roots: CubeRoot[] = [...dirSet].map((dir) => ({ dir, source: { type: 'dir', dir } }));
const { packages, errors } = resolveCubePackages(config.cubePackages);
for (const pkg of packages) {
for (const dir of pkg.dirs) {
roots.push({ dir, source: { type: 'package', packageName: pkg.name, dir } });
}
}
return { roots, errors };
}
/**
* The directories {@link findCubeRoots} would scan.
*
* Kept for callers that only want the paths; anything that needs to attribute
* a cube to where it came from should use `findCubeRoots` instead, which also
* reports the errors this one drops.
*
* @returns Array of absolute paths to directories containing cubes
*/
export function findCubeDirectories(): string[] {
return findCubeRoots().roots.map((root) => root.dir);
}
/**
@@ -56,10 +112,12 @@ interface CubeCandidate {
manifest: Manifest;
dir: string;
deployScript: string;
source: CubeSource;
}
/** What one root directory contributed. */
interface ScanResult {
root: CubeRoot;
candidates: CubeCandidate[];
errors: string[];
}
@@ -99,11 +157,23 @@ async function scanDirectory(currentDir: string, result: ScanResult): Promise<vo
manifest.id = cubeId;
manifest.schema = manifest.schema ?? z.object({});
// A `secrets` entry naming a key that is not in the schema protects
// nothing, and a typo in one is invisible at runtime — the value would
// just be persisted. Cheaper to refuse the cube than to ship the leak.
const unknown = (manifest.secrets ?? []).filter((key) => !(key in manifest.schema.shape));
if (unknown.length > 0) {
result.errors.push(
`Invalid manifest in ${manifestPath}: 'secrets' names ${unknown.join(', ')}, ` +
`which ${unknown.length === 1 ? 'is' : 'are'} not in the schema`
);
}
result.candidates.push({
id: cubeId,
manifest,
dir: currentDir,
deployScript: deployFile.name,
source: result.root.source,
});
}
} catch (err) {
@@ -118,9 +188,23 @@ async function scanDirectory(currentDir: string, result: ScanResult): Promise<vo
}
}
/** The message a duplicate id produces. Aborts the run — see `nopy.main.ts`. */
/**
* The message a duplicate id produces. Aborts the run — see `nopy.main.ts`.
*
* There is deliberately no precedence rule to fall back on: two cubes claiming
* one id are mutually exclusive, and the fix belongs upstream. So the message
* has to carry everything needed to go and make it, which means naming every
* claimant and how each got into the run.
*/
function duplicateError(id: string, group: CubeCandidate[]): string {
const where = group.map((c) => ` ${c.dir}`).join('\n');
const label = (candidate: CubeCandidate) =>
candidate.source.type === 'package' ? `package ${candidate.source.packageName}` : 'directory';
const width = Math.max(...group.map((candidate) => label(candidate).length));
const where = group
.map((candidate) => ` ${label(candidate).padEnd(width)} ${candidate.dir}`)
.join('\n');
return (
`Duplicate cube id '${id}' from ${group.length} sources:\n${where}\n` +
`Rename one of them, or remove a source from .nopyrc.json.`
@@ -137,20 +221,21 @@ function duplicateError(id: string, group: CubeCandidate[]): string {
* run, which is what makes the hard error testable.
*/
export async function loadCubes(): Promise<LoadResult> {
const cubesFolders = findCubeDirectories();
const { roots, errors: rootErrors } = findCubeRoots();
registerResolveHook();
const scans = await Promise.all(
cubesFolders.map(async (folder) => {
const result: ScanResult = { candidates: [], errors: [] };
if (fs.existsSync(folder)) {
await scanDirectory(folder, result);
roots.map(async (root) => {
const result: ScanResult = { root, candidates: [], errors: [] };
if (fs.existsSync(root.dir)) {
await scanDirectory(root.dir, result);
}
return result;
})
);
// Promise.all preserves input order regardless of completion order.
const errors = scans.flatMap((scan) => scan.errors);
const errors = [...rootErrors, ...scans.flatMap((scan) => scan.errors)];
// One directory reachable from two roots (a `cubeDirs` entry nested under a
// `.npcubes` marker, say) is one cube seen twice, not a collision.
@@ -172,7 +257,7 @@ export async function loadCubes(): Promise<LoadResult> {
// The map is still populated for the callers that only report; a duplicate
// is fatal, so which candidate landed here never reaches a deploy.
const [first] = group;
cubes[id] = new Cube(first.manifest, first.dir, first.deployScript);
cubes[id] = new Cube(first.manifest, first.dir, first.deployScript, first.source);
}
return { cubes, errors };
packages/nopy/src/cubes/packages.ts
+108
View File
@@ -0,0 +1,108 @@
/**
* Resolving cube packages named in `cubePackages` to directories on disk.
* @module cubes/packages
*/
import fs from 'node:fs';
import { createRequire } from 'node:module';
import path from 'node:path';
import type { CubePackageRef } from '../nopy.config.js';
/** An installed cube package, located and validated. */
export interface CubePackage {
/** The name it was requested under. */
name: string;
/** Absolute path to the package root. */
root: string;
/** Absolute paths to its cube directories, from `nopy.cubes`. */
dirs: string[];
}
/**
* Finds a package root without going through its `exports` map.
*
* `exports` is deliberately bypassed: a cube bundle ships directories, not an
* entry point, and requiring it to declare one would make the contract heavier
* for no gain. Reading `package.json` off disk also sidesteps pnpm's layout —
* `existsSync` follows the symlink pnpm plants at `node_modules/<name>`, which
* a directory scan would skip (`readdir` reports it as a symlink, not a
* directory).
*/
function findPackageRoot(ref: CubePackageRef): string | undefined {
// createRequire needs a file path, not a directory; the file need not exist.
const req = createRequire(path.join(ref.from, 'noop.js'));
for (const dir of req.resolve.paths(ref.spec) ?? []) {
if (fs.existsSync(path.join(dir, ref.spec, 'package.json'))) {
return path.join(dir, ref.spec);
}
}
return undefined;
}
/**
* Resolves every named package to its cube directories.
*
* Anything wrong is an error rather than a silent skip: naming a package in
* `cubePackages` is a statement that cubes are expected from it, and errors
* abort the run (see `nopy.main.ts`).
*/
export function resolveCubePackages(refs: CubePackageRef[]): {
packages: CubePackage[];
errors: string[];
} {
const packages: CubePackage[] = [];
const errors: string[] = [];
// `mergeValue` only de-duplicates arrays of primitives, and these are
// objects, so the same package named by a parent and a child config arrives
// twice. Last wins: configs merge root-first, so the last occurrence came
// from the most specific config and carries the right resolution origin.
const unique = new Map<string, CubePackageRef>();
for (const ref of refs) unique.set(ref.spec, ref);
for (const ref of unique.values()) {
const root = findPackageRoot(ref);
if (!root) {
errors.push(`Cube package '${ref.spec}' is not installed (looked up from ${ref.from}).`);
continue;
}
let manifest: { nopy?: { cubes?: unknown } };
try {
manifest = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf-8'));
} catch (err) {
errors.push(`Cube package '${ref.spec}': cannot read ${root}/package.json: ${err}`);
continue;
}
const declared = manifest.nopy?.cubes;
if (
!Array.isArray(declared) ||
declared.length === 0 ||
!declared.every((entry) => typeof entry === 'string')
) {
errors.push(
`Cube package '${ref.spec}' declares no cubes. ` +
`Expected "nopy": { "cubes": ["./cubes"] } in ${root}/package.json.`
);
continue;
}
const dirs: string[] = [];
for (const entry of declared as string[]) {
const dir = path.resolve(root, entry);
if (dir !== root && !dir.startsWith(root + path.sep)) {
errors.push(`Cube package '${ref.spec}': '${entry}' points outside the package.`);
} else if (!fs.existsSync(dir)) {
errors.push(`Cube package '${ref.spec}': '${entry}' does not exist in ${root}.`);
} else {
dirs.push(dir);
}
}
if (dirs.length > 0) packages.push({ name: ref.spec, root, dirs });
}
return { packages, errors };
}
packages/nopy/src/cubes/resolve-hook.mjs
+62
View File
@@ -0,0 +1,62 @@
/**
* A module resolve hook that lets a hand-written cube import the packages nopy
* itself already has.
*
* A manifest is loaded with `import(manifestPath)`, so its imports resolve from
* its own directory. A cube sitting in an arbitrary `cubeDirs` entry — no
* package.json above it, no node_modules beside it — therefore cannot import
* `@bitsquare/nopy-cube` or `zod` at all, and the run dies on
* ERR_MODULE_NOT_FOUND before a single deploy is built.
*
* A published cube bundle never reaches this: it declares its own dependencies
* and Node resolves them normally. This is for the local tree.
*
* Plain `.mjs` rather than TypeScript because the hook runs on its own thread,
* loaded by Node directly from `dist` — there is no compile step in that path.
*
* @module cubes/resolve-hook
*/
import { createRequire } from 'node:module';
import { pathToFileURL } from 'node:url';
/**
* The specifiers worth rescuing: what a manifest legitimately needs and cannot
* be expected to install for itself. Anything else stays a hard failure — a
* cube that wants a library should depend on it.
*/
const FALLBACK_ROOTS = ['@bitsquare/nopy-cube', '@bitsquare/nopy', 'zod'];
/** @type {NodeRequire | undefined} */
let fallbackRequire;
/**
* @param {{ from: string }} data - a URL inside the running CLI's own package,
* which is where the fallback resolution starts from.
*/
export function initialize(data) {
fallbackRequire = createRequire(data.from);
}
/** True for `zod` and for subpaths like `zod/v4` or `@bitsquare/nopy/package.json`. */
function isCovered(specifier) {
return FALLBACK_ROOTS.some((root) => specifier === root || specifier.startsWith(`${root}/`));
}
export async function resolve(specifier, context, next) {
try {
// Normal resolution first, always. A consumer that has its own copy
// installed keeps using it, so the hook can never introduce version skew —
// it only fills in for a lookup that was going to fail.
return await next(specifier, context);
} catch (error) {
if (!fallbackRequire || !isCovered(specifier)) throw error;
try {
return { url: pathToFileURL(fallbackRequire.resolve(specifier)).href, shortCircuit: true };
} catch {
// The CLI cannot see it either. Report the original failure, which names
// the importer rather than the CLI.
throw error;
}
}
}
packages/nopy/src/index.ts
+6 -8
View File
@@ -6,6 +6,9 @@
// Cubes module
export * from './cubes/index.js';
export type { Assignment, Origin, TVariables, Value } from './nopy.common.js';
// Variables
export { MASK, Variable, Variables } from './nopy.common.js';
export type {
ExecutionConfig,
HistoryConfig,
@@ -28,6 +31,8 @@ export type {
// Executor
export {
executeDeployCalls,
maskCommand,
maskVariables,
outputExecutionPlan,
summarizeResults,
} from './nopy.executor.js';
@@ -60,14 +65,7 @@ export {
} from './nopy.prompts.js';
export type { AuthSession, CubeSession, NopySession } from './nopy.session.js';
// Session management
export {
createSession,
filterInternalVariables,
listSessions,
loadSession,
saveSession,
separateEnvAndCubeVariables,
} from './nopy.session.js';
export { createSession, listSessions, loadSession, saveSession } from './nopy.session.js';
export type { WorkflowOptions, WorkflowResult } from './nopy.workflow.js';
// Workflow
export {
packages/nopy/src/nopy.common.ts
+188 -33
View File
@@ -1,49 +1,204 @@
/**
* Environment variable configuration
* Variable assignment and provenance
* @module nopy.common
*/
export type TVariables = Record<string, string | number | boolean>;
export namespace Variables {
export type ArtefactId = string;
export type Scope = 'defaults' | 'prompts' | 'params';
/** What a cube variable can hold — the value types `--data KEY=VALUE` can carry. */
export type Value = string | number | boolean;
/** A flat bag of variable values, keyed by name. */
export type TVariables = Record<string, Value>;
/**
* Where a value came from, in ascending precedence.
*
* The order is the point. It used to be implied by the field order of an object
* literal inside `Variables.get()` — load-bearing, invisible, and one careless
* reformat away from silently changing which value wins. Here it is stated once,
* in {@link RANK}, and everything else derives from it.
*
* - `default` — a `.default()` on the cube's schema
* - `env` — the `env` block of `.nopyrc.json`
* - `session` — read back from a recorded session on replay
* - `prompt` — what the user typed
* - `param` — handed over by a dependency spec or a hook's `exec()`
*/
export type Origin = 'default' | 'env' | 'session' | 'prompt' | 'param';
const RANK: Record<Origin, number> = {
default: 0,
env: 1,
session: 2,
prompt: 3,
param: 4,
};
/** One value handed to a variable, and where it came from. */
export interface Assignment {
value: Value;
origin: Origin;
}
export class Variables {
/** @summary env as configured in cube or session script */
defaults: Record<Variables.ArtefactId, TVariables> = {};
/** @summary env as configured via prompts */
prompts: Record<Variables.ArtefactId, TVariables> = {};
/** @summary env as handed via params (on hook calls) */
params: Record<Variables.ArtefactId, TVariables> = {};
/** What a secret shows as wherever a value would otherwise be printed. */
export const MASK = '********';
constructor(readonly global: TVariables = {}) {}
/**
* One variable of one cube, and every value it has ever been given.
*
* Two orderings are kept, deliberately: {@link assignments} is the raw trace in
* the order things happened, and {@link ordered} re-ranks it by origin. The
* first answers "how did we get here", the second answers "what wins".
*/
export class Variable {
/** Every assignment received, newest first. Never reordered. */
readonly assignments: Assignment[] = [];
assign(artefactId: Variables.ArtefactId, scope: Variables.Scope, values: TVariables = {}) {
if (!this[scope][artefactId]) {
this[scope][artefactId] = values;
} else {
Object.assign(this[scope][artefactId], values);
}
/**
* Declared a secret by the cube's manifest: kept out of saved sessions and
* masked wherever the value would otherwise be printed.
*/
redacted = false;
constructor(
readonly cube: string,
readonly name: string,
first: Assignment
) {
this.assign(first);
}
assign(assignment: Assignment): void {
this.assignments.unshift(assignment);
}
/**
* Merges the scopes for one cube, lowest precedence first:
* schema defaults → global `env` → prompts (or replayed session values) →
* params handed over by a dependency or a hook.
* The trace re-ranked by origin, winner first.
*
* Defaults sit at the bottom so `env` in `.nopyrc.json` can steer a run that
* never prompts (`--use-defaults`); a key that a dependency supplies is never
* prompted for, so prompts and params do not compete in practice.
* Stability is load-bearing here. The trace is newest-first and
* `Array.prototype.sort` is stable per spec, so two assignments sharing an
* origin keep their relative order and the newer one stays in front: the
* second dependency to pass a param wins, and the one it displaced is still
* visible underneath instead of being overwritten out of existence.
*/
get(artefactId: Variables.ArtefactId, scope?: Variables.Scope): TVariables {
if (scope) {
return this[scope][artefactId] || {};
}
get ordered(): Assignment[] {
return [...this.assignments].sort((a, b) => RANK[b.origin] - RANK[a.origin]);
}
/** The assignment that wins. Never undefined — a Variable is born with one. */
get effective(): Assignment {
return this.ordered[0];
}
get value(): Value {
return this.effective.value;
}
get origin(): Origin {
return this.effective.origin;
}
/** Safe to log: a redacted variable never yields its value. */
toJSON(): { cube: string; name: string; value: Value; origin: Origin } {
return {
...this.defaults[artefactId],
...this.global,
...this.prompts[artefactId],
...this.params[artefactId],
cube: this.cube,
name: this.name,
value: this.redacted ? MASK : this.value,
origin: this.origin,
};
}
}
/**
* Every variable of every cube in one run, with its provenance.
*/
export class Variables {
private readonly store: Record<string, Record<string, Variable>> = {};
private readonly secrets: Record<string, Set<string>> = {};
constructor(readonly env: TVariables = {}) {}
/**
* Marks keys of one cube as holding secrets.
*
* Retroactive as well as prospective, so it does not matter whether the
* caller declares before or after the values arrive.
*/
declareSecrets(cube: string, keys: readonly string[]): void {
this.secrets[cube] ??= new Set<string>();
const declared = this.secrets[cube];
for (const key of keys) declared.add(key);
for (const variable of this.all(cube)) {
if (declared.has(variable.name)) variable.redacted = true;
}
}
isSecret(cube: string, name: string): boolean {
return this.secrets[cube]?.has(name) ?? false;
}
/** Records values for one cube, all at the same origin. */
assign(cube: string, origin: Origin, values: TVariables = {}): void {
const bucket = this.bucket(cube);
for (const [name, value] of Object.entries(values)) {
const existing = bucket[name];
if (existing) existing.assign({ value, origin });
else bucket[name] = this.create(cube, name, { value, origin });
}
}
/** Every variable known for one cube. */
all(cube: string): Variable[] {
return Object.values(this.store[cube] ?? {});
}
/** One variable, or `undefined` if nothing has ever assigned to it. */
of(cube: string, name: string): Variable | undefined {
return this.store[cube]?.[name];
}
/** The effective values for one cube — what goes on the pyinfra command line. */
get(cube: string): TVariables {
const values: TVariables = {};
for (const variable of this.all(cube)) values[variable.name] = variable.value;
return values;
}
/**
* The effective values minus anything declared secret — what a session
* records. A secret is left out entirely rather than masked, so a replay sees
* it as absent and asks for it again.
*/
persistable(cube: string): TVariables {
const values: TVariables = {};
for (const variable of this.all(cube)) {
if (!variable.redacted) values[variable.name] = variable.value;
}
return values;
}
private create(cube: string, name: string, first: Assignment): Variable {
const variable = new Variable(cube, name, first);
variable.redacted = this.isSecret(cube, name);
return variable;
}
/**
* A cube's bucket, seeded on creation with the config `env`.
*
* `env` applies to every cube, so it becomes a real assignment on each of them
* rather than a parallel bag merged in at read time. That is what lets it
* carry an origin, show up in the trace, and lose to a prompt by the same rule
* as everything else.
*/
private bucket(cube: string): Record<string, Variable> {
const existing = this.store[cube];
if (existing) return existing;
const bucket: Record<string, Variable> = {};
this.store[cube] = bucket;
for (const [name, value] of Object.entries(this.env)) {
bucket[name] = this.create(cube, name, { value, origin: 'env' });
}
return bucket;
}
}
packages/nopy/src/nopy.executor.ts
+47 -11
View File
@@ -3,9 +3,10 @@
* @module nopy.executor
*/
import type { DependencySpec } from '@bitsquare/nopy-cube';
import { getLogger } from '@logtape/logtape';
import { execa } from 'execa';
import type { DependencySpec } from './cubes/types.js';
import { MASK } from './nopy.common.js';
const log = getLogger(['nopy', 'executor']);
@@ -23,10 +24,47 @@ export interface DeployCall {
command: string[];
/** Environment variables for the cube */
env: Record<string, unknown>;
/** Schema keys the cube's manifest declared as secrets */
secrets?: string[];
/** Cube dependencies */
dependencies: DependencySpec[];
}
/**
* The command as it is safe to show: the SSH password, and every `--data KEY=…`
* whose key the manifest declared a secret, have their values replaced.
*
* pyinfra takes its data on the command line, so the real values have to be in
* `call.command` — this is the last point before they would reach a log, a
* `--print-only` dump or a dry-run plan.
*/
export function maskCommand(call: DeployCall): string {
const command = call.command.join(' ');
const quoteMeta = (key: string) => key.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
// The builder always quotes a `--data` value, so the closing quote bounds it.
const masked = (call.secrets ?? []).reduce(
(acc, key) => acc.replace(new RegExp(`(--data "${quoteMeta(key)}=)[^"]*"`, 'g'), `$1${MASK}"`),
command
);
return masked.replace(/(--password )\S+/g, `$1${MASK}`);
}
/**
* The cube's variables as they are safe to show.
*
* This used to guess, masking any key whose name contained "password" — which
* missed `TOKEN` and `PSK`, and was defeated anyway by the unmasked command
* printed on the line above it. The manifest says which keys are secret now.
*/
export function maskVariables(call: DeployCall): Record<string, string> {
const secrets = new Set(call.secrets ?? []);
return Object.fromEntries(
Object.entries(call.env).map(([key, value]) => [key, secrets.has(key) ? MASK : String(value)])
);
}
/**
* Result of executing a deployment command
*/
@@ -73,7 +111,7 @@ async function executeCall(call: DeployCall): Promise<ExecutionResult> {
try {
log.info(`Executing: ${call.cube} -> ${call.host}`);
log.debug(`Command: ${commandStr}`);
log.debug(`Command: ${maskCommand(call)}`);
// Inherit stdio for live output
await execa({ shell: true })(commandStr, {
@@ -112,8 +150,8 @@ export function outputExecutionPlan(calls: DeployCall[], asJson?: boolean): void
const plan = calls.map((call) => ({
cube: call.cube,
host: call.host,
command: call.command.join(' '),
variables: call.env,
command: maskCommand(call),
variables: maskVariables(call),
}));
console.log(JSON.stringify({ plan }, null, 2));
return;
@@ -124,15 +162,13 @@ export function outputExecutionPlan(calls: DeployCall[], asJson?: boolean): void
for (let i = 0; i < calls.length; i++) {
const call = calls[i];
console.log(`Step ${i + 1}: ${call.cube} -> ${call.host}`);
console.log(` Command: ${call.command.join(' ')}`);
console.log(` Command: ${maskCommand(call)}`);
const envKeys = Object.keys(call.env);
if (envKeys.length > 0) {
const variables = maskVariables(call);
if (Object.keys(variables).length > 0) {
console.log(' Variables:');
for (const [key, value] of Object.entries(call.env)) {
// Mask sensitive values
const displayValue = key.toLowerCase().includes('password') ? '********' : String(value);
console.log(` ${key}=${displayValue}`);
for (const [key, value] of Object.entries(variables)) {
console.log(` ${key}=${value}`);
}
}
console.log();
packages/nopy/src/nopy.main.ts
+11 -3
View File
@@ -8,7 +8,12 @@ import { BuildContext } from './cubes/dependencies.js';
import { loadCubes } from './cubes/index.js';
import { Variables } from './nopy.common.js';
import { getConfigPaths, loadConfig } from './nopy.config.js';
import { type ExecutionResult, executeDeployCalls, summarizeResults } from './nopy.executor.js';
import {
type ExecutionResult,
executeDeployCalls,
maskCommand,
summarizeResults,
} from './nopy.executor.js';
import { addToHistory, DEFAULT_HISTORY_SIZE } from './nopy.history.js';
import { type NopySession, saveSession } from './nopy.session.js';
import { runWorkflow } from './nopy.workflow.js';
@@ -72,6 +77,9 @@ function printActiveConfig(
if (config.hosts.length > 0) lines.push(` Hosts: ${config.hosts.join(', ')}`);
if (config.cubeDirs.length > 0) lines.push(` Cube dirs: ${config.cubeDirs.join(', ')}`);
if (config.cubePackages.length > 0) {
lines.push(` Cube pkgs: ${config.cubePackages.map((ref) => ref.spec).join(', ')}`);
}
if (opts.continueOnError) lines.push(' Execution: continue-on-error');
const envEntries = Object.entries(config.env);
@@ -185,7 +193,7 @@ export async function nopy(opts: NopyOptions = {}): Promise<NopyResult | undefin
const sessionForSaving: NopySession = {
...workflow.session,
cubes: context.cubeSessions,
env: variables.get('global'),
env: config.env,
};
if (saveSessionPath && !workflow.isReplay) {
@@ -203,7 +211,7 @@ export async function nopy(opts: NopyOptions = {}): Promise<NopyResult | undefin
console.log('\n Deploy Commands\n ───────────────\n');
for (const call of context.deployCalls) {
console.log(` # ${call.cube} -> ${call.host}`);
console.log(` ${call.command.join(' ')}\n`);
console.log(` ${maskCommand(call)}\n`);
}
return {
success: true,
packages/nopy/src/nopy.prompts.ts
+28 -11
View File
@@ -36,11 +36,17 @@ function suggestCubes(input: string | undefined, choices: CubeChoice[]): CubeCho
export async function CubeSelection(
cubes: Record<string, Cube>
): Promise<{ selectedCubes: string[] }> {
// The package a cube came from is part of the label rather than a separate
// column: `suggest` filters on the label, so typing a package name narrows
// the list to that bundle.
const cubeChoices: CubeChoice[] = Object.values(cubes)
.sort((a, b) => a.id.localeCompare(b.id))
.map((cube) => ({
name: cube.id,
message: `${cube.id} - ${cube.name}`,
message:
cube.source.type === 'package'
? `${cube.id} - ${cube.name} (${cube.source.packageName})`
: `${cube.id} - ${cube.name}`,
}));
// Clear terminal and move cursor to top
@@ -176,24 +182,35 @@ interface FormChoice {
initial: string;
}
/**
* Asks the user for a cube's variables and records the answers.
*
* Reads what to offer out of `variables`, so the caller is expected to have
* assigned the schema defaults first — which `BuildContext.resolveCube` does.
* Deliberately not falling back to `cube.getDefaults()` here: calling it a
* second time re-evaluates every lazily declared default, so a cube generating
* one would show a different value than the one the run had already recorded.
*/
export async function VariableAssignment<S extends AnyObjectSchema>(
cube: Cube<S>,
variables: Variables
variables: Variables,
opts: { keys?: string[] } = {}
) {
const schema = cube.manifest.schema.shape;
const defaults = cube.getDefaults() as Record<string, unknown>;
const params = variables.get(cube.id, 'params');
const resolved = variables.get(cube.id);
const variablesToConfigure: Record<string, unknown> = {};
// Every schema key is offered, not just the ones carrying a `.default()` — a
// field without one is precisely the field that has to be asked about. Keys a
// dependency or hook already supplied are left alone. The value shown is the
// Every schema key is offered by default, not just the ones carrying a
// `.default()` — a field without one is precisely the field that has to be
// asked about. `opts.keys` narrows that to a subset, which is how a replay
// asks only about the gaps it cannot fill itself.
//
// A key a dependency or hook supplied is left alone. The value shown is the
// one the run would otherwise use, so `env` from `.nopyrc.json` is visible
// (and editable) rather than silently overridden by whatever is typed.
for (const key of Object.keys(schema)) {
if (params[key] !== undefined) continue;
variablesToConfigure[key] = resolved[key] ?? defaults[key];
for (const key of opts.keys ?? Object.keys(schema)) {
if (variables.of(cube.id, key)?.origin === 'param') continue;
variablesToConfigure[key] = resolved[key];
}
if (Object.keys(variablesToConfigure).length === 0) return;
@@ -217,7 +234,7 @@ export async function VariableAssignment<S extends AnyObjectSchema>(
const zodType = schema[key];
coercedResult[key] = zodType ? coerceValue(value, zodType) : value;
}
variables.assign(cube.id, 'prompts', coercedResult);
variables.assign(cube.id, 'prompt', coercedResult);
} catch {
// User cancelled
}

Some files were not shown because too many files have changed in this diff Show More