diff --git a/docs/performance-analysis.md b/docs/performance-analysis.md new file mode 100644 index 00000000..435bc391 --- /dev/null +++ b/docs/performance-analysis.md @@ -0,0 +1,423 @@ +# Performance Analysis & Optimization Opportunities + +**Date**: 2026-03-07 +**Scope**: Full-stack performance analysis — backend PTY handling, SSE broadcasting, frontend terminal rendering, local echo overlay, DOM updates, config/scaling limits. +**Constraint**: All recommendations preserve existing functionality including local echo, backpressure, anti-flicker pipeline, and mobile support. + +--- + +## Executive Summary + +The codebase is already well-optimized in critical paths. The multi-layer backpressure system, adaptive terminal batching, DEC 2026 sync markers, and incremental state serialization are strong. The main opportunities are in **reducing unnecessary work** (SSE filtering, DOM rebuilds, lazy terminal init) rather than algorithmic changes. + +**Top 5 high-impact opportunities:** + +| # | Optimization | Impact | Risk | Effort | +|---|-------------|--------|------|--------| +| 1 | Session-scoped SSE subscriptions | Bandwidth -60-80%, CPU -40% | Medium | Medium | +| 2 | Lazy xterm.js for minimized subagent windows | Memory -3.5MB at 50 agents | Low | Low | +| 3 | Targeted badge update (skip full tab rebuild) | Eliminates O(n) reflow on badge change | Low | Low | +| 4 | Conditional SSE padding (tunnel-only, terminal-only) | Bandwidth -70% when tunneled | Low | Low | +| 5 | Canvas renderer on mobile | GPU pressure reduction, battery savings | Low | Low | + +--- + +## 1. SSE Broadcasting + +### Current State +- **92 event types** broadcast to all connected clients (max 100) +- Single `JSON.stringify()` per event, shared across all clients (efficient) +- **No per-client filtering** — every client receives every event regardless of which session they're viewing +- 8KB padding appended to **every** event when tunnel is active (forces Cloudflare proxy flush) +- Backpressure: clients marked as backpressured if `reply.raw.write()` returns false; recovery via `session:needsRefresh` + +### Bottlenecks + +**B1: No session-scoped SSE subscriptions** (`server.ts:1986`) +- Client viewing session A still receives all events for sessions B through T +- With 20 active sessions, ~95% of terminal events are irrelevant to any given client +- Cost: wasted bandwidth, CPU for JSON parsing, and event handler dispatch on client + +**B2: Unconditional 8KB padding** (`server.ts:1977`) +- Every event gets 8KB comment padding when tunnel is active +- A `task:updated` event (~200 bytes payload) becomes ~8.2KB +- High-frequency events like `session:terminal` need the padding; low-frequency events like `session:created` don't + +### Recommendations + +**R1: Session-scoped SSE subscriptions** (High impact) +- Add `?sessions=id1,id2` query param to `/api/events` SSE endpoint +- Server filters events by session ID before broadcasting +- Client subscribes to active session + "global" events (session lifecycle, system) +- Re-subscribes on tab switch (or subscribe to all with client-side filter as fallback) +- **Savings**: ~80% bandwidth reduction for single-session viewers; ~60% for multi-session dashboards + +**R2: Tiered SSE padding** (Medium impact) +- Only pad `session:terminal` events and SSE heartbeats (the two that need proxy flush) +- Skip padding for low-frequency structural events (`session:created`, `task:updated`, etc.) +- **Savings**: ~70% padding overhead reduction; terminal events already large enough to flush + +--- + +## 2. Terminal Rendering + +### Current State (Well-Optimized) +- **6-layer anti-flicker pipeline**: Server batching (adaptive 16-50ms) → DEC 2026 sync wrap → single JSON serialize → client rAF batching → sync segment parser → chunked buffer loading (32KB/frame) +- **64KB/frame write budget** with DEC 2026 sync-segment awareness (prevents 141KB single-frame freezes) +- **3-layer backpressure**: SSE cap (128KB queued → drop + refresh), frame budget (64KB/frame), chunked restore (32KB/frame) +- WebGL renderer enabled by default with canvas fallback on context loss +- Typical latency: 16-32ms; worst case: ~115ms (50ms server batch + 50ms sync wait + 16ms rAF) + +### Bottlenecks + +**B3: WebGL on mobile** (`app.js:627-637`) +- Mobile GPUs are weaker; WebGL context loss more likely on low-end devices +- Canvas renderer is sufficient for mobile (typically 1 session, smaller viewport) + +**B4: Static scrollback for all sessions** (`app.js:572`) +- Default 5000 lines scrollback for all sessions regardless of activity level +- Heavy output sessions (build logs, test runners) accumulate large scroll buffers + +**B5: No addon lazy loading** +- FitAddon, Unicode11Addon, and WebGLAddon all loaded at terminal init +- Unicode11Addon only needed for CJK content; WebGLAddon is large + +### Recommendations + +**R3: Force canvas renderer on mobile** (Low risk) +- Detect `MobileDetection.isMobile()` and skip WebGL addon loading +- Reduces GPU memory pressure, prevents context loss crashes +- Mobile typically has 1-2 sessions — canvas performance is more than adequate + +**R4: Dynamic scrollback based on session activity** (Low risk) +- Active sessions (working state): 5000 lines (current default) +- Inactive/idle sessions: reduce to 2000 lines +- Restore on session select (fetch from server buffer) +- **Savings**: ~60% scrollback memory for idle sessions + +**R5: Lazy-load Unicode11Addon** (Low risk) +- Only load when CJK content is detected in terminal output +- Detection: check for characters in CJK Unicode ranges during ANSI stripping (already iterating) +- Most sessions never need it + +--- + +## 3. DOM & Session Tab Rendering + +### Current State +- Session tabs use **intelligent incremental updates** with debounced 100ms rendering +- Incremental path: only updates changed properties (classes, textContent, badges) when session list is stable +- Full rebuild path: triggered when sessions added/removed **or badge count changes** +- Subagent windows: per-window xterm.js instances, even when minimized + +### Bottlenecks + +**B6: Badge count change triggers full tab rebuild** (`app.js:3207-3209`) +- A single subagent badge increment on one tab triggers `_fullRenderSessionTabs()` — rebuilds entire sidebar HTML via `innerHTML =` +- With 20 sessions, this is an O(n) reflow for a single badge number change +- Badge changes are frequent during active subagent work + +**B7: Minimized subagent windows retain xterm.js instances** (`subagent-windows.js`) +- 50 subagent windows × ~75KB per xterm.js instance = ~3.75MB DOM memory +- Minimized windows are invisible but their terminals remain in DOM +- xterm.js instances continue processing resize events even when hidden + +**B8: `backdrop-filter: blur()` on overlays** (`styles.css:2246-2247, 3098`) +- Forces new stacking context, disables browser compositing optimizations +- 50-100ms layout thrashing on modal open/close +- Only 2 uses, but they're on frequently toggled overlays + +### Recommendations + +**R6: Targeted badge update without full rebuild** (Low risk) +- When badge count changes but session list is stable, update only the badge `` textContent +- Keep incremental path for badge changes; only use full rebuild for structural changes (add/remove sessions) +- **Savings**: Eliminates O(n) reflow per badge change; reduces to O(1) targeted update + +**R7: Lazy xterm.js initialization for subagent windows** (Medium impact) +- Only create xterm.js Terminal instance when window is restored/maximized +- On minimize: serialize terminal buffer, dispose Terminal instance, keep buffer in memory +- On restore: create new Terminal, write buffer back +- **Savings**: ~3.5MB DOM reduction at 50 minimized agents; eliminates hidden resize processing +- **Trade-off**: ~200-500ms restore delay (buffer write), mitigated by chunked loading + +**R8: Replace `backdrop-filter: blur()` with `background: rgba()`** (Low risk) +- Use semi-transparent background instead of blur effect +- Or use `will-change: transform` hint if blur is kept +- **Savings**: Eliminates forced recomposition layer; 50-100ms faster overlay open + +--- + +## 4. Backend PTY & State Management + +### Current State (Excellent) +- **BufferAccumulator**: Array-based chunking with lazy join on read — avoids O(n) string concatenation +- **ANSI stripping**: Throttled at 150ms intervals with lazy evaluation (not per-chunk) +- **State persistence**: 500ms debounce + incremental JSON caching per session (only dirty sessions re-serialized) +- **Expensive parsers**: Throttled to 150ms window, accumulated data capped at 64KB +- **Memory**: All buffers have hard limits (2MB terminal, 1MB text, 1000 messages, 64KB line buffer) + +### Bottlenecks + +**B9: Pending clean data cap at 64KB** (`session.ts:1097-1133`) +- Between 150ms processing windows, raw PTY data accumulates in `_pendingCleanData` +- Capped at 64KB — excess data rolls off (old data discarded) +- During heavy output (large build logs), this means parsers may miss content +- Acceptable trade-off for performance, but worth documenting + +**B10: `LRUMap.delete()` is O(n) worst case** (`utils/lru-map.ts:137-138`) +- When deleting the newest entry, iterates all keys to find new newest +- Rare in practice (delete is uncommon; set/get are hot paths) +- Could matter during mass cleanup of 500 agents + +### Recommendations + +**R9: Consider adaptive pending data cap** (Low priority) +- During idle detection (critical to get right), increase cap to 128KB +- During active working state, keep at 64KB (parsers less critical) +- **Benefit**: More accurate idle detection during heavy output + +**R10: Track second-newest in LRUMap** (Low priority) +- Maintain a `_secondNewestKey` alongside `_newestKey` +- On delete of newest, promote second-newest without iteration +- Only matters at scale (500+ agents with frequent eviction) + +--- + +## 5. Local Echo & Input Path + +### Current State (Well-Designed) +- **DOM overlay approach** — `` elements in `.xterm-screen` at z-index 7, completely independent of `terminal.write()` +- **Render caching**: `_lastRenderKey` includes text, position, column offsets — skips redundant re-renders +- **Input flow**: Char accumulation → Enter triggers flush → 80ms delay before `\r` (ensures text reaches PTY first) +- **Tab completion**: Baseline snapshot → detect buffer change → 300ms fallback timer +- **CJK support**: Per-character width detection with `terminal.unicode.getStringCellWidth()` preferred, manual fallback +- **Prompt detection**: Bottom-up line scan, O(rows) — cached position, column-lock prevents jitter + +### Bottlenecks + +**B11: tmux send-keys latency** (~50-100ms per input) +- Each `writeViaMux()` spawns a child process (`tmux send-keys`) +- Text and Enter sent separately with 50ms delay between +- For rapid typing: characters batch before Enter, so overhead is per-command not per-keystroke +- **Acceptable trade-off** for session persistence (tmux survives server restarts) + +**B12: 80ms delay between text flush and Enter** (`app.js:872-875`) +- Intentional: ensures text reaches PTY before Enter, preventing Ink from processing empty input +- Adds 80ms to perceived Enter-to-response latency +- Could potentially be reduced with acknowledgment-based approach + +**B13: Scroll listener on terminal viewport** (`zerolag-input-addon.ts:139`) +- 50ms debounced re-render on scroll — acceptable but fires frequently during heavy output +- Overlay hidden when scrolled up (correct behavior), shown when at bottom + +### Recommendations + +**R11: Reduce Enter delay from 80ms to 50ms** (Low risk, test carefully) +- The tmux `send-keys` already has 50ms internal delay +- Combined with network latency, 80ms client-side may be excessive +- Test with Ink-heavy sessions (Claude Code's status bar) — if text arrives before Enter at 50ms, reduce +- **Savings**: 30ms perceived latency reduction per command + +**R12: Batch tmux send-keys via stdin pipe** (Medium effort, high impact for rapid input) +- Instead of spawning `tmux send-keys` per input, maintain a persistent connection +- Use `tmux -C` (control mode) for programmatic interaction without child process spawning +- **Savings**: Eliminate ~50-100ms process spawn overhead per input +- **Risk**: Control mode has different semantics; needs careful testing with session persistence + +**R13: Skip overlay re-render during heavy output scroll** (Low risk) +- When terminal is receiving >10KB/s output, hide overlay entirely (user isn't typing during heavy output) +- Re-show overlay after 500ms of output silence +- **Savings**: Eliminates unnecessary DOM overlay re-renders during build logs / test output + +--- + +## 6. Polling & File Watchers + +### Current State +- **SubagentWatcher**: 1s base poll, full scan throttled to every 5s, fs.watch() on known directories +- **TranscriptWatcher**: 1 per session, fs.watch() primary with 1s poll fallback +- **ImageWatcher**: chokidar per session with 100ms stability poll, burst limit 20/10s +- **TeamWatcher**: chokidar primary with 30s poll fallback, LRU caches (50 teams, 200 tasks) +- **RalphTracker**: Todo cleanup every 5 minutes + +### Scaling Profile (20 sessions) +| Component | Instances | Frequency | Total ops/sec | +|-----------|-----------|-----------|---------------| +| SubagentWatcher | 1 (global) | Full scan every 5s | 0.2/s | +| TranscriptWatcher | 20 | 1s poll (fallback) | 20/s max | +| ImageWatcher | 20 | 100ms poll (during writes only) | 200/s burst | +| TeamWatcher | 1 (global) | 30s poll (fallback) | 0.03/s | +| SSE heartbeat | 1 (global) | 15s | 0.07/s | +| SSE dead client check | 1 (global) | 30s | 0.03/s | +| Mux stats collection | 1 (global) | 2s | 0.5/s | +| **Total steady-state** | | | **~21/s** | + +### Recommendations + +**R14: Increase TranscriptWatcher poll interval to 2s** (Low risk) +- Transcript changes are infrequent (new messages every few seconds at most) +- fs.watch() is the primary mechanism; polling is fallback +- **Savings**: Halves fallback filesystem checks (20/s → 10/s for 20 sessions) + +**R15: Share chokidar instances for co-located session directories** (Medium effort) +- Sessions in the same parent directory could share a single chokidar watcher with depth:3 +- Common case: multiple sessions in `~/projects/foo/` — one watcher covers all +- **Savings**: Reduce chokidar instances from 20 to ~5-10 for typical workloads + +--- + +## 7. Frontend Asset Delivery + +### Current State +- **app.js**: 12,027 lines (source) → esbuild minified → gzip/brotli compressed (~30-40KB gzipped) +- **Static caching**: `maxAge: '1y'` via `@fastify/static` +- **Service worker**: Push notification handler only — no asset caching +- **No code splitting**: Single monolithic app.js bundle + +### Bottlenecks + +**B14: No cache-busting mechanism** +- `maxAge: '1y'` means browsers cache aggressively +- After deployment, users need `Ctrl+Shift+R` to see updates +- No content hash in filenames or ETags for automatic invalidation + +**B15: Monolithic app.js** +- All 12K lines loaded on initial page load regardless of which features are used +- Ralph wizard, plan orchestrator UI, team management — all loaded upfront +- Mobile loads the same bundle as desktop + +### Recommendations + +**R16: Add content hash to asset filenames** (Medium impact) +- Build step: rename `app.js` → `app.[hash].js` +- Generate a manifest or inject hash into HTML template +- Keep `maxAge: '1y'` — cache invalidation happens via filename change +- **Savings**: Eliminates stale cache issues after deployment; removes need for manual hard refresh + +**R17: Code-split app.js into core + feature modules** (High effort, medium impact) +- Core (~4K lines): terminal, SSE, session management, tabs, input handling +- Deferred (~8K lines): Ralph wizard, plan UI, team management, subagent windows, image viewer +- Load deferred modules on first use via dynamic `import()` or lazy `