mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-10-07 16:09:43 +02:00
feat: vendor gesture-control source into packages/gesture-control
Bring the Ark0N/codeman-gesture-control repo in-tree as the codeman-gesture-control workspace package so the hand-tracking overlay can be developed in the Codeman repo. New npm run build:gesture bundles src/codeman/entry.ts into the served gesture-codeman.js; scripts/build.mjs reruns it on every production build. Source formatted to Codeman's prettier style. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
arkon
@@ -0,0 +1,224 @@
|
||||
# Codeman Gesture Control — Prototype Build Plan
|
||||
|
||||
> Canonical spec for this project. A Jarvis-style hand-tracking input layer for
|
||||
> the Codeman dashboard. Webcam sees your hands; you pinch-drag session tabs
|
||||
> between Screen columns and fire discrete gesture commands. Runs entirely
|
||||
> in-browser, camera feed never leaves the machine.
|
||||
|
||||
Build it as a **standalone prototype first** (its own folder, fake tabs) so the
|
||||
input feel can be validated on real hardware before any integration with
|
||||
Codeman's existing drag/command code.
|
||||
|
||||
---
|
||||
|
||||
## Goal & success criteria
|
||||
|
||||
Build a `GestureController` module + a self-contained demo page that:
|
||||
|
||||
1. Opens the webcam and runs MediaPipe Gesture Recognizer at ~30fps.
|
||||
2. Emits a smoothed cursor position and a `pinch` state (grab/release) from hand landmarks.
|
||||
3. Lets you **drag fake tabs between columns by pinching**, dropping on release.
|
||||
4. Fires **discrete gesture commands** (open palm, thumbs up, victory) onto an event bus.
|
||||
5. Feels responsive — drag lag is not perceptible, jitter is filtered out.
|
||||
|
||||
**Done when:** you can sit at your desk, pinch a tab, move it to another column,
|
||||
release, and it lands — reliably, without visible jitter, with the camera
|
||||
mounted at your chosen angle.
|
||||
|
||||
---
|
||||
|
||||
## Tech stack
|
||||
|
||||
- **MediaPipe Tasks Vision** (`@mediapipe/tasks-vision`) — `GestureRecognizer` in `VIDEO` running mode, `numHands: 1` for v1 (add 2 later). Loads the prebuilt `gesture_recognizer.task` model + WASM from CDN.
|
||||
- **Vanilla TS + Vite** for the prototype (no framework needed; keep it portable so the module drops into Codeman regardless of its stack). If Codeman is React, the module stays framework-agnostic and you wrap it in a hook at integration time.
|
||||
- **One-Euro filter** for cursor smoothing — implement it directly, it's ~40 lines and is the correct tool for noisy interactive landmark streams (low lag at speed, heavy smoothing when still).
|
||||
- **Web Worker** for inference is a **Phase 4** optimization — do NOT start there. Get it working on the main thread first; only move to a worker if the dashboard UI stutters.
|
||||
|
||||
---
|
||||
|
||||
## File structure
|
||||
|
||||
```
|
||||
gesture-proto/
|
||||
├── index.html # demo page: video preview + columns of fake tabs
|
||||
├── package.json
|
||||
├── vite.config.ts
|
||||
├── src/
|
||||
│ ├── main.ts # wires GestureController -> demo UI
|
||||
│ ├── gesture/
|
||||
│ │ ├── GestureController.ts # core: camera + recognizer + state machine + events
|
||||
│ │ ├── OneEuroFilter.ts # cursor smoothing
|
||||
│ │ ├── pinch.ts # pinch detection w/ hysteresis
|
||||
│ │ ├── landmarks.ts # landmark index constants + helpers
|
||||
│ │ └── types.ts # event payload types, config
|
||||
│ └── demo/
|
||||
│ ├── tabs.ts # fake tab/column model + render
|
||||
│ └── overlay.ts # draws hand skeleton + cursor dot over video (debug)
|
||||
└── README.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core algorithms (the parts that decide whether it feels good)
|
||||
|
||||
### 1. Cursor from landmarks
|
||||
The drag cursor is the **midpoint of thumb tip (landmark 4) and index tip (landmark 8)**, in normalized [0,1] coords from MediaPipe.
|
||||
|
||||
- **Mirror X** (`x = 1 - x`) — the webcam image is flipped relative to the user.
|
||||
- Map normalized → screen pixels against the dashboard's bounding rect.
|
||||
- Run the resulting (x, y) through **two independent One-Euro filters** (one per axis) before using it. Raw landmarks jitter by several pixels even when the hand is still; this is the single most important quality step.
|
||||
|
||||
### 2. Pinch detection with hysteresis
|
||||
Compute euclidean distance between landmark 4 and landmark 8. **Normalize by hand size** (e.g. distance wrist→middle-finger-MCP, landmarks 0→9) so the threshold is robust to how close the hand is to the camera.
|
||||
|
||||
- Use **two thresholds, not one** (hysteresis): enter pinch below `PINCH_ON` (e.g. 0.35 of hand size), exit only above `PINCH_OFF` (e.g. 0.5). This stops flickering between grab/release at the boundary — critical for not "dropping" a tab mid-drag.
|
||||
- Require the pinch state to persist N frames (e.g. 2–3) before firing, to reject single-frame noise.
|
||||
|
||||
### 3. State machine
|
||||
```
|
||||
IDLE ──hand detected──> HOVER ──pinch on──> GRABBED ──pinch off──> (drop) ──> HOVER
|
||||
^ | |
|
||||
└────hand lost───────────┴────────────────hand lost────────────────────────┘
|
||||
```
|
||||
- `HOVER`: cursor moves, highlights the tab/column under it (hit-test).
|
||||
- `GRABBED`: the grabbed tab follows the cursor; emit `drag` events.
|
||||
- On `pinch off` in GRABBED: hit-test cursor against drop columns, emit `drop {tabId, targetColumnId}` or `dropCancelled` if outside any column.
|
||||
|
||||
### 4. Discrete gestures → command bus
|
||||
From `result.gestures[0].categoryName`, debounced (fire once per gesture entry, not every frame while held):
|
||||
- `Open_Palm` held ~1s → `command: "halt-all"` (dead-man's-switch — pauses every session; genuinely useful for autonomous loops).
|
||||
- `Thumb_Up` → `command: "approve"`.
|
||||
- `Victory` → `command: "new-session"`.
|
||||
- Map these to the SAME command names your voice layer already dispatches, so both input sources converge on one dispatcher.
|
||||
|
||||
---
|
||||
|
||||
## GestureController public API (target shape)
|
||||
|
||||
```ts
|
||||
const gc = new GestureController({
|
||||
video: videoEl,
|
||||
surface: dashboardEl, // coords mapped against this element's rect
|
||||
numHands: 1,
|
||||
pinchOn: 0.35, pinchOff: 0.5,
|
||||
palmHoldMs: 1000,
|
||||
});
|
||||
|
||||
gc.on("hover", ({ x, y, targetId }) => {...});
|
||||
gc.on("grab", ({ x, y }) => {...});
|
||||
gc.on("drag", ({ x, y }) => {...}); // throttled to frame rate
|
||||
gc.on("drop", ({ targetColumnId }) => {...});
|
||||
gc.on("command",({ name }) => {...}); // halt-all | approve | new-session
|
||||
gc.on("status", ({ fps, handPresent, pinchDist }) => {...}); // debug HUD
|
||||
|
||||
await gc.start(); // requests camera, loads model
|
||||
gc.stop();
|
||||
```
|
||||
|
||||
Keep it **transport-agnostic**: it emits semantic events, it does NOT know about
|
||||
Codeman's DOM. Integration is just subscribing to these events and calling
|
||||
Codeman's existing tab-move / command functions.
|
||||
|
||||
---
|
||||
|
||||
## Phased build (each phase is independently testable — stop and feel it before moving on)
|
||||
|
||||
**Phase 0 — Scaffold & camera (½ day)**
|
||||
Vite + TS project. `index.html` with a mirrored `<video>` and a "start" button (camera must be a user gesture). Confirm `getUserMedia` works and you see yourself. Must be served over http(s), not `file://`.
|
||||
|
||||
**Phase 1 — Recognizer + debug overlay (½ day)**
|
||||
Load `GestureRecognizer` (`VIDEO` mode, CDN model+wasm). Run `recognizeForVideo(video, performance.now())` in a `requestAnimationFrame` loop. Draw the 21-point skeleton + an FPS counter on a canvas over the video. **Checkpoint: confirm you're getting ≥25fps on your actual camera/lighting setup.** Tune lighting here.
|
||||
|
||||
**Phase 2 — Cursor + pinch (1 day)**
|
||||
Implement `OneEuroFilter` and `pinch.ts`. Render a cursor dot driven by the filtered thumb/index midpoint. Show live pinch distance in the HUD and a color change on grab. **Checkpoint: the dot is steady when your hand is still, and pinch grab/release is crisp with no flicker.** Tune filter constants (`minCutoff`, `beta`) and pinch thresholds here — this is where the "feel" is won or lost.
|
||||
|
||||
**Phase 3 — Drag the fake tabs (1 day)**
|
||||
Build `tabs.ts`: 3 columns of draggable fake "sessions." Wire the state machine: hover-highlight, grab, drag-follow, drop-with-hit-test. **Checkpoint: you can move a tab across columns reliably 10/10 times.** This is the core demo and the real go/no-go for the whole idea.
|
||||
|
||||
**Phase 4 — Discrete commands + polish (1 day)**
|
||||
Add gesture→command bus with debouncing and the 1s open-palm halt. Add an on-screen toast when a command fires. Optional: move inference to a Web Worker if the UI stutters; add second-hand support.
|
||||
|
||||
**Phase 5 — Codeman integration (✅ working, 2026-06-07)**
|
||||
Drop `gesture/` into Codeman via a new consumer `src/codeman/entry.ts` (the core is unchanged; the demo's `main.ts` is *not* the integration point). It binds `grab`/`drag`/`drop` to real `.session-tab`s (grab-to-detach → `app.detachSession`) and pinch-taps the Run / Run Shell toolbar buttons. Runs in Codeman behind `CODEMAN_GESTURE=1`. See the detailed status under "Implementation status" below.
|
||||
|
||||
> **Prerequisite (decided 2026-06-06, ✅ done 2026-06-07): Codeman tab-detach first.**
|
||||
> Codeman needed a **tab-detach / undock** feature — a session pops out into its
|
||||
> own browser window — *before* gesture wiring, because gestures can only drag DOM
|
||||
> *within* the one page that owns the camera (you can't drag a node across isolated
|
||||
> tabs/OS windows). So undock is a Codeman session-placement op the gesture `drop`
|
||||
> *triggers*. **Shipped** as `app.detachSession(id)` → `/session/:id` solo window +
|
||||
> BroadcastChannel sync + re-dock on close (the gesture layer calls it directly).
|
||||
> Per-monitor placement via `getScreenDetails` stays in the multi-monitor backlog.
|
||||
|
||||
---
|
||||
|
||||
## Tuning defaults to start from (then adjust by feel)
|
||||
- One-Euro: `minCutoff ≈ 1.0`, `beta ≈ 0.01` (raise `beta` if drag lags during fast moves; lower `minCutoff` if it's jittery when still).
|
||||
- Pinch: `PINCH_ON 0.35`, `PINCH_OFF 0.5` (fractions of hand-size reference distance).
|
||||
- `min_detection_confidence` / `min_tracking_confidence` ≈ 0.6; lower if quick gestures get missed, raise if you get false hands.
|
||||
- Camera: target 30–60fps, even frontal lighting on the hand zone (matters more than the sensor).
|
||||
|
||||
---
|
||||
|
||||
## Hardware note
|
||||
|
||||
Prototype on the **MacBook M1 Max built-in cam** for zero-friction Phase 0–3.
|
||||
For the real setup, switch to **iPhone via Continuity Camera** mounted at desk
|
||||
level aimed at your hand-gesture zone — best sensor + best angle. Decide
|
||||
front-facing (pointing/pinch-to-grab) vs overhead (swipe/drag-on-a-plane) mount
|
||||
before Phase 3, since it slightly changes the gesture grammar.
|
||||
|
||||
---
|
||||
|
||||
## Implementation status (kept current)
|
||||
|
||||
- ✅ Phase 0, ✅ Phase 1 — see top-level `CLAUDE.md` and `gesture-proto/README.md`.
|
||||
- ✅ Phase 1 fps checkpoint — 60fps on MacBook + iPhone 17 Pro (Continuity Camera).
|
||||
- ✅ Phase 2 — `OneEuroFilter.ts` + `pinch.ts`: filtered cursor dot + pinch hysteresis. Cursor + `pinchDist` + `pinching` on the `status` event; HUD shows pinch distance, cursor ring turns green on grab.
|
||||
- ✅ Phase 3 — `demo/tabs.ts`: 3 Screen columns of draggable session tabs. Controller owns the per-hand pinch state machine and emits `grab`/`drag`/`drop` in surface pixels (with a `hand` id); the demo hit-tests and moves tabs. Two-handed (drag two tabs at once); drop-on-vanish releases a tab if a pinched hand leaves frame.
|
||||
- ✅ Beyond plan — two-hand tracking (`numHands: 2`, filters keyed by handedness) and a live camera picker.
|
||||
- ✅ Camera (2026-06-06) — front-facing iPhone 17 Pro main lens (Chrome) is the default. **The superwide / Desk View (ultra-wide, overhead) camera now works with no issues and tracking is confirmed on it** — the earlier "Safari-only / stretched / unusable" finding is superseded. Pick either via the in-app camera picker.
|
||||
- ✅ Phase 4 (built) — `commands.ts`: debounced gesture→command bus. `Thumb_Up`→approve, `Victory`→new-session (edge-triggered, fire once per entry), `Open_Palm` held `palmHoldMs`→halt-all (dead-man's-switch) with a 0–1 `haltProgress` charge surfaced on `status`. Commands ignore a pinching (mid-drag) hand.
|
||||
- ⛔ **Phase 4 unwired in the demo (2026-06-06).** User wants pinch-drag only — an open palm while reaching to pinch kept charging the hold-to-halt. `main.ts` no longer subscribes to `command`/`haltProgress` and the command/charge toasts are gone. The `GestureController` core is untouched and still emits both events, so Phase 5 (or a re-enabled demo) can pick them up unchanged.
|
||||
- 🐛 **Drag-position fix (2026-06-06).** A `.dragging` tab is `position: absolute`; the `.column`s establish a containing block via `backdrop-filter`, so board-local left/top were offset by the column's own position — tabs in the middle/right columns flew to the right on grab. Fix: `tabs.ts` reparents the floating tab onto `#board` (no filter/transform) for the drag, so the coordinates `moveTo` computes match the containing block.
|
||||
- ✅ **Phase 5 — WORKING at the desk (2026-06-07).** Prerequisite cleared: Codeman tab-detach/undock works in the runtime (`app.detachSession(id)` is the idempotent hook). The gesture overlay runs live in the real Codeman dashboard on `:5000` and was confirmed by the user (normal tab): fullscreen cam + hand/cursor tracking, undock-by-pinch, and Run/Run Shell taps.
|
||||
- **Integration shape: in-page overlay, built into Codeman beta.** The gesture `core` (`src/gesture/`) ships **unchanged**; the consumer is `src/codeman/entry.ts`, esbuild-bundled (`npm run build:codeman`) and served by Codeman at `/gesture/gesture-codeman.js`. A full-viewport, click-through overlay maps coords straight to `elementFromPoint`.
|
||||
- **Gestures (routed by what the pinch lands on):** (a) **grab → in-page floating panel** *(⚠️ pivoted 2026-06-08 — was grab-to-detach)*: pinch a `.session-tab`, a ghost clone follows the hand, pull >`DETACH_PULL_PX` (70) and release → `floatSession(id, x, y)` spawns a re-grabbable `.cg-float` iframe of `/session/:id` (640×420). This **replaced** `window.app.detachSession(id)` (an OS window is a sealed box the hand can't move again — a one-way trip); the float stays in-page so the hand keeps control. See `../../docs/MULTIMONITOR_DESIGN.md`. (b) **Run / Run Shell taps** — pinch over `#runBtn`→`app.run()` / `.btn-shell`→`app.runShell()` and release in place; drift >`TAP_CANCEL_PX` (45) cancels. `CLICK_SELECTOR` is the extensible list. (c) **Fullscreen dimmed cam** by default, **⛶** toggles corner PiP.
|
||||
- **Self-hosted MediaPipe (no CDN).** `entry.ts` passes `wasmBase: "/gesture/wasm"` + `modelUrl: "/gesture/gesture_recognizer.task"`; Codeman serves them same-origin. The CDN path failed in the normal browser tab (content/ad blocker blocking `jsdelivr`/`googleapis`) → surfaced as `failed: {"isTrusted":true}` once `entry.ts` learned to report non-Error throws. Core also gained a GPU→CPU delegate fallback.
|
||||
- **Gated by `CODEMAN_GESTURE=1` (OFF by default).** Under the flag Codeman injects the module script (dashboard only, not `/session/:id` solo popups), cache-busts it with `?v=<mtime>` (static is `max-age=1y`), and widens CSP (`'wasm-unsafe-eval'` + `worker-src 'self' blob:`; same-origin assets now covered by `'self'`). Flag off ⇒ Codeman HTML/CSP unchanged.
|
||||
- **Codeman-side / version control:** the detach + instance isolation + base gesture overlay are committed on `Ark0N/Codeman` branch `beta/session-detach`, open as **PR #103** (tip `afea6d6`; `ceca853` after I fixed its `auth.ts` format:check → CI green). Gotcha: the local prod clone `~/.codeman/app` tracks only `master`, so the branch is hidden until `git fetch origin beta/session-detach` (this briefly misled me into a bogus local reconstruction `03b31b8`, since deleted). The session improvements — direct-detach via `window.app.detachSession`, Run/Run Shell pinch-taps, self-hosted MediaPipe (`/gesture/wasm` + `.task`), and the `server.ts` mtime cache-bust — were **ported onto PR #103** in commit `eea84db` (CI green); their source is `Ark0N/codeman-gesture-control` (`src/codeman/entry.ts`).
|
||||
- **Commits (gesture-proto):** `ddf9cda` (consumer: detach/cam/errors) → `2dd97da` (detach direct) → `21ef793` (Run/Run Shell taps) → `e055b79` (self-host MediaPipe).
|
||||
- **Next:** tune feel; optional in-strip reorder (deferred — user chose detach-only) and more buttons (Stop). Discrete `command` events remain available but unwired (pinch-only). Hand-off brief: `../docs/CODEMAN_DETACH_BRIEF.md`.
|
||||
|
||||
## Backlog (requested, for later)
|
||||
|
||||
- ✅ **Fullscreen mode** — done. Toggle button fullscreens the `#stage`;
|
||||
`:fullscreen` CSS fills the viewport and the coord mapping adapts since it
|
||||
reads the stage rect every frame.
|
||||
- ✅ **Multi-monitor mode — A+C BUILT & validated at the desk (2026-06-08).** The
|
||||
eventual real goal (fling a Codeman session onto an external display by gesture)
|
||||
is reached. **Design → [`../../docs/MULTIMONITOR_DESIGN.md`](../../docs/MULTIMONITOR_DESIGN.md)**,
|
||||
approach **A+C**:
|
||||
- **A — in-page floating panels** (`581fcf9`, `3e0447a`): `entry.ts`
|
||||
`floatSession(id, x, y)` pops a tab into a re-grabbable `.cg-float` iframe of
|
||||
`/session/:id` (640×420) instead of `window.app.detachSession`. The session
|
||||
stays in the camera-owning page's DOM, so the hand keeps control — fixing the
|
||||
one-way-trip flaw of OS-window detach.
|
||||
- **C-span — spanned window** (`063fd8f`, `59946b8`): `scripts/span-codeman.sh`
|
||||
launches a Brave-first (`BROWSER=` override) `--app` window sized to the
|
||||
display union; prereq macOS "Displays have separate Spaces" OFF + re-login.
|
||||
One-click via the **Codeman header button** → `POST /api/system/span-displays`
|
||||
(PR #103 `95b0035`). **Validated:** one window spans both monitors and a panel
|
||||
drags across the seam.
|
||||
- **C-snap** (`getScreenDetails` snapping + seam dead-band) and the **re-dock**
|
||||
gesture/zone are **still pending** — not needed for basic cross-seam dragging.
|
||||
- **Concurrent-rendering question** was RESOLVED first: Codeman already mounts
|
||||
live terminals into floating panels (teammate terminals, log-viewer SSE
|
||||
windows, the iframe-able `/session/:id` solo route), so the live float needed
|
||||
no new Codeman rendering.
|
||||
|
||||
Note: the public event surface evolved from the original API sketch. The
|
||||
controller stays transport-agnostic but emits coordinate-only `grab`/`drag`/
|
||||
`drop` (hit-testing lives in the consumer, since only it knows the DOM/columns).
|
||||
`hover`/`dropCancelled`/`targetId` were dropped; hover highlighting is derived
|
||||
from the `status` snapshot instead.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Feature brief: Session tab detach / undock (for Codeman)
|
||||
|
||||
> ✅ **SHIPPED — on GitHub as PR #103 (open).** Branch `beta/session-detach` on
|
||||
> `Ark0N/Codeman` (base `master`): "feat(web): session detach/undock + beta
|
||||
> instance isolation (port 5000)", containing detach/undock + instance isolation
|
||||
> + the base gesture overlay (commit `afea6d6`). `app.detachSession(id)` in
|
||||
> `app.js` opens `/session/:id` as a solo window (another live client of the same
|
||||
> session), tracks it (badge + `BroadcastChannel` sync + re-dock on close), and is
|
||||
> the single idempotent entry point both the on-tab ⧉ icon and the gesture layer
|
||||
> call. PTY fan-out (the open question below) resolved **yes**, so no streaming
|
||||
> work was needed. CI green after I fixed a prettier format:check on `auth.ts`
|
||||
> (commit `ceca853`).
|
||||
>
|
||||
> ⚠️ **Note:** the local **prod** clone `~/.codeman/app` only tracks `master`, so
|
||||
> the PR branch is invisible there until `git fetch origin beta/session-detach`.
|
||||
> (Earlier today I briefly mis-concluded the PR didn't exist and made a bogus
|
||||
> local reconstruction — deleted. The PR was real all along.) The gesture-side
|
||||
> *improvements* from this session — direct-detach, Run/Run Shell pinch-taps,
|
||||
> self-hosted MediaPipe, `server.ts` cache-bust — were **ported onto PR #103**
|
||||
> (commit `eea84db`, CI green); their source is `Ark0N/codeman-gesture-control`.
|
||||
> The rest of this doc is the original hand-off brief, kept for history.
|
||||
|
||||
> Hand-off brief for **Codeman** to refine and implement **on a beta branch**.
|
||||
> Authored from the gesture-control project, which needs this as a prerequisite.
|
||||
> Codeman is "aicodeman": a Fastify + WebSocket server streaming xterm.js
|
||||
> terminal (tmux) sessions to a web dashboard.
|
||||
|
||||
## Goal
|
||||
|
||||
Let a session "tab" pop out of the main dashboard into its **own browser
|
||||
window** (and back). Each detached window shows just that one session's
|
||||
terminal, fully live. This is a standalone UX win *and* a prerequisite for
|
||||
gesture control later (a hand-gesture "drop" will eventually trigger
|
||||
detach/relocate — but that's a separate project; **this feature is plain UI
|
||||
buttons only**).
|
||||
|
||||
## Core approach (refine as needed)
|
||||
|
||||
- Add a **"Detach" control** on each tab. It opens a new browser window
|
||||
(`window.open`) pointing at a **single-session view** — ideally a real route
|
||||
like `/session/:id` so the popup just loads a URL and attaches like a normal
|
||||
client.
|
||||
- The detached window runs its **own xterm.js instance connected to the same
|
||||
session's WebSocket**, so it's live, not a screenshot.
|
||||
- Keep the dashboard and detached windows **in sync** (session list, titles,
|
||||
alive/dead state, focus) — via the existing events channel, or a
|
||||
`BroadcastChannel` if simpler.
|
||||
- Support **re-dock** (close popup → tab returns to the dashboard) and handle the
|
||||
popup being closed/refreshed gracefully.
|
||||
|
||||
## The one critical question to resolve first (in Codeman's own code)
|
||||
|
||||
Can the server currently **fan out one session's PTY/tmux output to multiple
|
||||
concurrent WebSocket clients**, or is it single-consumer? A detached window is a
|
||||
*second* viewer of the same session. If it's single-consumer today, that's the
|
||||
main change: make the pty→socket stream **broadcast to N subscribers** (and merge
|
||||
input) so dashboard + popup can both watch/type. This likely matters more than
|
||||
the UI work.
|
||||
|
||||
## Other decisions for Codeman
|
||||
|
||||
- Per-session route (`/session/:id`) vs. a single-page popup that's told which id
|
||||
to show.
|
||||
- Multi-monitor placement later via the Window Management API
|
||||
(`getScreenDetails`) — **out of scope now**, just don't design against it.
|
||||
- Auth/cookie sharing so a popup window authenticates the same as the dashboard.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Implement on a **beta branch**, not `main`.
|
||||
- The gesture-control side keeps a **read-only** copy of Codeman (its `.git`
|
||||
removed); the live `Ark0N/Codeman` repo is **not** touched from here. Codeman
|
||||
implements this itself.
|
||||
|
||||
## Why this is sequenced before gesture wiring
|
||||
|
||||
Gestures can only drag DOM **within the single page that owns the camera**; you
|
||||
cannot drag a node across isolated browser tabs / OS windows. So undock must be a
|
||||
**Codeman session-placement operation** that a gesture `drop` later *triggers* —
|
||||
not something the gesture layer does. Detach first; wire gestures to it after.
|
||||
@@ -0,0 +1,260 @@
|
||||
# Multi-monitor gesture design — in-page panels + spanned window (A + C)
|
||||
|
||||
**Status:** **A + C-span BUILT & validated at the desk (2026-06-08).** C-snap
|
||||
(`getScreenDetails` snapping) still pending. Decided + built 2026-06-08.
|
||||
**Supersedes** the OS-window detach as the *gesture* verb (see "Why detach
|
||||
broke movability") — *now actually replaced in `entry.ts`, not just planned.*
|
||||
**Companion docs:** `CODEMAN_DETACH_BRIEF.md` (the original window.open detach),
|
||||
`../gesture-proto/docs/BUILD_PLAN.md` (canonical build spec, Phase 5).
|
||||
|
||||
> ## Implementation status (2026-06-08)
|
||||
> - ✅ **A — in-page floating panels.** `entry.ts` `floatSession(id, x, y)` spawns
|
||||
> a `.cg-float` div with an `<iframe src="/session/:id">` (640×420) at the drop
|
||||
> point; panels are re-grabbable. Replaces `window.app.detachSession`. Commits
|
||||
> `581fcf9`, `3e0447a`.
|
||||
> - ✅ **C-span — spanned window.** `scripts/span-codeman.sh` (Brave-first;
|
||||
> `BROWSER=` override) launches a `--app` window sized to the display union.
|
||||
> Commits `063fd8f`, `59946b8`. **Validated at the desk:** one window spans
|
||||
> both monitors (3432×1080) and a panel drags across the seam.
|
||||
> - ✅ **Launch entry point in Codeman.** A header "multi-monitor" button (replaces
|
||||
> the notification bell) → `POST /api/system/span-displays` → spawns the span
|
||||
> script. Bundled `span-codeman.sh` into Codeman's repo. **PR #103** `95b0035`.
|
||||
> - ✅ **Cache-bust** all same-origin module scripts/CSS (`renderIndexHtml` →
|
||||
> `cacheBustAssets`), so frontend edits show on a normal reload. PR #103 `b5ea711`.
|
||||
> - ⏳ **C-snap** (`getScreenDetails` snapping + seam dead-band) — not built; not
|
||||
> required for basic cross-seam dragging. Also pending: the re-dock gesture/zone.
|
||||
> - **Known caveats from the desk run:** dead band on a taller/offset external
|
||||
> display (inherent to one spanning rect); superwide lens not selectable in the
|
||||
> fresh span-window browser profile; 3-monitor works unchanged but dead-space +
|
||||
> cursor-sensitivity caveats grow.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Pinch a Codeman session, drag it anywhere — including **across a second
|
||||
physical monitor** — drop it, and have it stay where you put it and stay
|
||||
grabbable again. The "fling a session onto the external display by gesture"
|
||||
end-goal from the BUILD_PLAN backlog, made real **without losing the ability to
|
||||
move a session after you've placed it.**
|
||||
|
||||
## The root constraint (why this design, not the others)
|
||||
|
||||
The hand only exists in **the one page that owns the camera**. Hand tracking,
|
||||
the cursor, and `document.elementFromPoint` hit-testing all live in that single
|
||||
document. **An OS window created by `window.open` is a sealed box that page
|
||||
cannot reach into** — no shared DOM, no shared cursor.
|
||||
|
||||
> **Why detach broke movability.** Today `entry.ts` → `detach(id)` calls
|
||||
> `window.app.detachSession(id)`, which `window.open`s the session into its own
|
||||
> OS window. The instant it leaves the camera-owning page, the hand can never
|
||||
> touch it again. Detach-by-pinch *works*, but it's a one-way trip.
|
||||
|
||||
**Rule:** anything you want to keep gesture-movable must stay inside **one
|
||||
page's DOM.** This design honors that with two composed pieces:
|
||||
|
||||
- **A — In-page floating panels.** "Detach" pops a session into a free-floating,
|
||||
absolutely-positioned element *in the same page* (not an OS window), so the
|
||||
hand keeps control forever.
|
||||
- **C — One window spanning both monitors.** Run Codeman in a single window
|
||||
stretched across both physical displays, so "drag across monitors" is just
|
||||
"drag across the page," and the second monitor's pixels are actually used.
|
||||
|
||||
Option **B** (one OS window per monitor + a distributed BroadcastChannel cursor
|
||||
protocol + cross-window session hand-off) was considered and deferred: it's the
|
||||
only path to *independent* per-monitor OS windows, but it's a much larger build
|
||||
and reintroduces the cross-window wall this design exists to avoid.
|
||||
|
||||
---
|
||||
|
||||
## Part A — In-page floating panels
|
||||
|
||||
### Codeman already has the primitive
|
||||
|
||||
`panels-ui.js` has a `.detached` "floating window": an absolutely-positioned
|
||||
`<div>` with `panel.style.top/left/width/height` and a drag handler
|
||||
(`setupMonitorDrag`) — **all in-page, no `window.open`.** The session-tab detach
|
||||
simply picked the wrong primitive (the OS-window one). Part A reuses the
|
||||
in-page one.
|
||||
|
||||
### Concurrent session rendering — RESOLVED (2026-06-08): live floats are feasible now
|
||||
|
||||
The main *dashboard view* renders **one active session at a time** (a single
|
||||
shared `this.terminal` opened into `#terminalContainer` in `terminal-ui.js:26`,
|
||||
swapped by `selectSession()`). But the page is **not** limited to one terminal —
|
||||
the PR #103 branch already ships **three independent in-page concurrent
|
||||
floating-content subsystems** we can reuse, so a live floating panel per session
|
||||
needs **no new Codeman rendering architecture**:
|
||||
|
||||
1. **Teammate terminals** (`panels-ui.js` `teammateTerminals` Map +
|
||||
`subagent-windows.js`). A `Map` of **concurrent live `new Terminal()`
|
||||
instances**, each `terminal.open(body)`'d into a floating panel, bound to a
|
||||
`sessionId` + tmux `paneTarget`, **seeded via REST** buffer fetch
|
||||
(`/api/sessions/:id/teammate-pane-buffer/:pane`), **input via REST**
|
||||
(`/api/sessions/:id/teammate-pane-input`), with lazy-mount (`_lazyPaneTarget`)
|
||||
and `dispose()` cleanup. *This is option 2 already built and shipping.*
|
||||
2. **Log-viewer windows** (`panels-ui.js:2632`). Concurrent draggable floating
|
||||
windows, each with its own `EventSource` **SSE stream** and lifecycle map —
|
||||
proof of the generic "floating window + independent per-window stream"
|
||||
pattern.
|
||||
3. **`/session/:id` solo route** (`server.ts:573`, `renderIndexHtml(soloId)`,
|
||||
`text/event-stream` at `:633`). A full **standalone live session page** —
|
||||
**iframe-able** — reusing the exact multi-client fan-out detach already
|
||||
depends on. Solo mode deliberately **omits** the gesture overlay
|
||||
(`server.ts:1010-1014`), so there's no nested-overlay problem.
|
||||
|
||||
**The PTY multi-viewer fan-out is confirmed** (the brief's open "yes"): a session
|
||||
is addressable by multiple concurrent clients — the solo route, the teammate
|
||||
REST endpoints, and the on-tab pop-out all view the same live session.
|
||||
|
||||
**Resolution:** skip the static-preview fallback. Build live floats directly,
|
||||
ranked by new-code cost:
|
||||
|
||||
- **Primary — iframe the solo route.** `floatPanel(id)` =
|
||||
`<div class="cg-float" data-id><iframe src="/session/:id"></iframe></div>`.
|
||||
The iframe is a complete live session view (its own terminal + stream client);
|
||||
the gesture layer moves the **div** and the iframe rides along. Lowest new
|
||||
code; reuses proven fan-out; no terminal wiring. Trade-off: a full app shell
|
||||
per float (heavier — fine for a few, watch memory at many).
|
||||
- **Richer alt — native teammate-style panel.** Mount a `new Terminal()` in the
|
||||
float body, seed via the session buffer endpoint, feed via the teammate
|
||||
stream. Native (no iframe), lighter per-float, same-document. Use if the
|
||||
iframe feels heavy or you want tighter integration.
|
||||
|
||||
Either way the hand only **places** the float; typing into it uses a keyboard
|
||||
(focus the iframe / native terminal) — consistent with "gesture places,
|
||||
keyboard types."
|
||||
|
||||
### Gesture grammar (replaces the current detach path in `entry.ts`)
|
||||
|
||||
The grab/drag/drop plumbing already exists; only the **drop action** changes.
|
||||
|
||||
- **Grab** a `.session-tab[data-id]` (unchanged: `onGrab`, ghost-follow).
|
||||
- **Pull** past `DETACH_PULL_PX` to arm (unchanged: `grab.armed`).
|
||||
- **Drop while armed** → **no longer** `window.app.detachSession`. Instead spawn
|
||||
an **in-page floating panel** for that session id at the drop point. New
|
||||
method `floatPanel(id, x, y)` replacing `detach(id)`.
|
||||
- **Re-grab** a floating panel (new `PANEL_SELECTOR`, e.g. `.cg-float[data-id]`)
|
||||
→ move it; drop anywhere → it stays. This is the capability detach lost.
|
||||
- **Drop a panel back over the tab strip** (or a dock zone) → **re-dock**
|
||||
(remove the float; session returns to a plain tab). Mirrors the `.detached` →
|
||||
attach toggle Codeman already has.
|
||||
- **Keep `window.open` detach as a separate, deliberate verb** — e.g. a button,
|
||||
or a distinct "throw up and off-screen" gesture — for intentionally parking a
|
||||
session in its own OS window. It is *not* the default pinch action anymore.
|
||||
|
||||
### `entry.ts` change surface
|
||||
|
||||
- New state: `floats: Map<string, FloatingPanel>` (id → element + position),
|
||||
parallel to the existing `grabs`/`taps` maps.
|
||||
- `onGrab`: extend hit-testing to also match `PANEL_SELECTOR`, so an existing
|
||||
float can be re-grabbed (priority: panel over tab when overlapping).
|
||||
- `onDrop`: replace `if (grab.armed) this.detach(grab.id)` with
|
||||
`this.floatPanel(...)`; add the panel re-dock branch.
|
||||
- `floatPanel(id, x, y)`: create/show the in-page panel (Part A option 1/2),
|
||||
position it absolutely at the drop point. Idempotent per id (re-grab moves the
|
||||
existing one, never duplicates).
|
||||
- Coordinate mapping is **already viewport-pixel based** (the click-through
|
||||
surface maps cursor → viewport px), so it needs **no change** for spanning —
|
||||
see Part C.
|
||||
|
||||
---
|
||||
|
||||
## Part C — One window spanning both monitors
|
||||
|
||||
Once the Codeman window physically covers both displays, Part A's panels drag
|
||||
across the seam for free, because the gesture cursor is already in viewport
|
||||
pixels and the viewport now spans both monitors.
|
||||
|
||||
### macOS setup (operational, near-zero code)
|
||||
|
||||
1. **System Settings → Desktop & Dock → uncheck "Displays have separate
|
||||
Spaces."** (Requires a logout/login.) This is what lets a single window
|
||||
straddle two physical displays.
|
||||
2. Run Codeman **maximized, not fullscreen.** Browser fullscreen is *per
|
||||
display* and will **not** span — use a maximized/borderless window dragged to
|
||||
cover both monitors. (A kiosk/`--app` Chrome window sized to the union rect is
|
||||
the cleanest.) **Automated by `scripts/span-codeman.sh`** — it reads the
|
||||
display-union rect (Finder desktop bounds) and launches a **Brave-first**
|
||||
Chromium-family `--app` window sized to it (fresh per-browser profile so the
|
||||
geometry flags are honored; `BROWSER=` overrides — plain Chrome bounced on the
|
||||
desk machine). One-click from the **Codeman header "multi-monitor" button**
|
||||
(`POST /api/system/span-displays`, which spawns this script), or run it
|
||||
directly. Step 1 + logout is still manual; the script warns if spanning isn't
|
||||
active.
|
||||
3. Arrange the two monitors as a contiguous rectangle in Display settings so the
|
||||
union has no vertical offset gap.
|
||||
|
||||
### Coordinate model & the bezel seam
|
||||
|
||||
- Enumerate displays with **`window.getScreenDetails()`** (Chrome, secure
|
||||
context, `window-management` permission). Gives each screen's
|
||||
`left/top/width/height/availLeft/...` in a **virtual-desktop coordinate
|
||||
space** spanning all monitors.
|
||||
- Use it for **screen-edge snapping zones**: e.g. dropping a panel within the
|
||||
right screen's bounds snaps it to fill that screen; the seam between the two
|
||||
screens' rects is a "halt / boundary" zone the cursor crosses.
|
||||
- **Account for the bezel gap.** The two monitors are physically separated, but
|
||||
the spanned window's pixels are contiguous — a panel dragged across the seam
|
||||
visually jumps the bezel. Optional: add a dead-band at the seam x-coordinate
|
||||
so a panel snaps to one side rather than straddling.
|
||||
- `getScreenDetails` is **only needed for snapping/zone logic**, not for basic
|
||||
dragging — dragging works the moment the window spans. So Part C can ship in
|
||||
two steps: (1) just span + free drag, (2) add `getScreenDetails` snapping.
|
||||
|
||||
### Constraints to surface to the user
|
||||
|
||||
- macOS-specific; the "separate Spaces" toggle is global and affects all apps.
|
||||
- Real fullscreen is unavailable (must run maximized).
|
||||
- A bezel-width discontinuity sits in the middle of the coordinate space.
|
||||
- `window-management` permission prompts once.
|
||||
|
||||
---
|
||||
|
||||
## Build phases
|
||||
|
||||
1. ✅ **A-MVP — in-page live float, single monitor (DONE, `581fcf9`/`3e0447a`).**
|
||||
`entry.ts` `floatSession(id, x, y)` spawns an **iframe of `/session/:id`** in a
|
||||
`.cg-float` div (640×420) at the drop point; re-grab + move work (re-dock zone
|
||||
still TBD). Movability restored — the live session rides inside the page. (The
|
||||
method is named `floatSession`, not the design-sketch `floatPanel`.)
|
||||
2. ✅ **C-span — span the window (DONE, `063fd8f`/`59946b8`; validated at desk).**
|
||||
macOS "separate Spaces" off + `scripts/span-codeman.sh` launches a maximized
|
||||
`--app` window across the display union (Brave-first; `BROWSER=` override).
|
||||
**Confirmed:** panels drag across the seam — viewport mapping held with zero
|
||||
`entry.ts` change. Launchable one-click from the **Codeman header button** →
|
||||
`POST /api/system/span-displays` (PR #103 `95b0035`), or by running the script.
|
||||
3. ⏳ **C-snap — screen-aware snapping (pending).** Add `getScreenDetails`; snap a
|
||||
panel to the screen it's dropped on; add the seam dead-band.
|
||||
4. **A-alt (optional).** Swap the iframe float for a native teammate-style
|
||||
`new Terminal()` panel if the iframe shell feels heavy or many floats strain
|
||||
memory.
|
||||
|
||||
## Open questions / verification before coding
|
||||
|
||||
- [x] **Concurrent rendering — RESOLVED (2026-06-08).** A live session view
|
||||
*can* be mounted into an arbitrary in-page container concurrently with the
|
||||
main view. The PR #103 branch already ships it three ways (teammate terminals,
|
||||
log-viewer SSE windows, the iframe-able `/session/:id` solo route); fan-out
|
||||
confirmed. Primary path: iframe the solo route. See the resolved section above.
|
||||
- [ ] **Native-panel live feed (only if taking A-alt, not the iframe):** trace
|
||||
the exact transport that pushes *continuous* teammate-pane output after the
|
||||
REST buffer seed (`pendingData` flush source). The iframe path sidesteps this.
|
||||
- [ ] **`window.app.detachSession` location:** when wiring the *kept* OS-window
|
||||
detach verb, note this method was **not found in PR #103 source** (only
|
||||
server-side `detachSessionListeners`); it works at runtime, so confirm where
|
||||
it's actually defined before extending it.
|
||||
- [ ] **Spanning feasibility on the actual desk setup** (monitor arrangement,
|
||||
whether "separate Spaces" off is acceptable to the user globally).
|
||||
- [ ] **Re-dock target:** decide the re-dock gesture/zone (drop over tab strip
|
||||
vs a dedicated dock region).
|
||||
|
||||
## Touch-points summary
|
||||
|
||||
| Layer | File | Change | Status |
|
||||
|-------|------|--------|--------|
|
||||
| Gesture consumer | `gesture-proto/src/codeman/entry.ts` | `detach()` → `floatSession()`; `floats` map; panel re-grab; (later) re-dock branch + `getScreenDetails` snapping | ✅ float+re-grab done (`581fcf9`/`3e0447a`); re-dock/snap pending |
|
||||
| Gesture core | `gesture-proto/src/gesture/*` | **none** — stays transport-agnostic | ✅ unchanged |
|
||||
| Codeman — launch | `src/web/routes/system-routes.ts`, `src/web/public/{index.html,panels-ui.js}`, `scripts/span-codeman.sh` | header button → `POST /api/system/span-displays` → spawn span script (bundled into repo) | ✅ PR #103 `95b0035` |
|
||||
| Codeman — caching | `src/web/server.ts` | `cacheBustAssets()` — `?v=<mtime>` on all same-origin `.js`/`.css` so frontend edits show on normal reload | ✅ PR #103 `b5ea711` |
|
||||
| Ops | macOS display settings + `scripts/span-codeman.sh` | "separate Spaces" off + re-login; Brave-first maximized spanning window | ✅ validated at desk |
|
||||
Reference in New Issue
Block a user