Codeman/packages/gesture-control/docs/MULTIMONITOR_DESIGN.md

Multi-monitor gesture design — in-page panels + spanned window (A + C)

Status: A + C-span BUILT & validated at the desk (2026-06-08). C-snap (getScreenDetails snapping) still pending. Decided + built 2026-06-08. Supersedes the OS-window detach as the gesture verb (see "Why detach broke movability") — now actually replaced in entry.ts, not just planned. Companion docs: CODEMAN_DETACH_BRIEF.md (the original window.open detach), ../gesture-proto/docs/BUILD_PLAN.md (canonical build spec, Phase 5).

Implementation status (2026-06-08)

• ✅ A — in-page floating panels. entry.ts floatSession(id, x, y) spawns a .cg-float div with an <iframe src="/session/:id"> (640×420) at the drop point; panels are re-grabbable. Replaces window.app.detachSession. Commits 581fcf9, 3e0447a.
• ✅ C-span — spanned window. scripts/span-codeman.sh (Brave-first; BROWSER= override) launches a --app window sized to the display union. Commits 063fd8f, 59946b8. Validated at the desk: one window spans both monitors (3432×1080) and a panel drags across the seam.
• ✅ Launch entry point in Codeman. A header "multi-monitor" button (replaces the notification bell) → POST /api/system/span-displays → spawns the span script. Bundled span-codeman.sh into Codeman's repo. PR #103 95b0035.
• ✅ Cache-bust all same-origin module scripts/CSS (renderIndexHtml → cacheBustAssets), so frontend edits show on a normal reload. PR #103 b5ea711.
• ⏳ C-snap (getScreenDetails snapping + seam dead-band) — not built; not required for basic cross-seam dragging. Also pending: the re-dock gesture/zone.
• Known caveats from the desk run: dead band on a taller/offset external display (inherent to one spanning rect); superwide lens not selectable in the fresh span-window browser profile; 3-monitor works unchanged but dead-space + cursor-sensitivity caveats grow.

---

Goal

Pinch a Codeman session, drag it anywhere — including across a second physical monitor — drop it, and have it stay where you put it and stay grabbable again. The "fling a session onto the external display by gesture" end-goal from the BUILD_PLAN backlog, made real without losing the ability to move a session after you've placed it.

The root constraint (why this design, not the others)

The hand only exists in the one page that owns the camera. Hand tracking, the cursor, and document.elementFromPoint hit-testing all live in that single document. An OS window created by window.open is a sealed box that page cannot reach into — no shared DOM, no shared cursor.

Why detach broke movability. Today entry.ts → detach(id) calls window.app.detachSession(id), which window.opens the session into its own OS window. The instant it leaves the camera-owning page, the hand can never touch it again. Detach-by-pinch works, but it's a one-way trip.

Rule: anything you want to keep gesture-movable must stay inside one page's DOM. This design honors that with two composed pieces:

• A — In-page floating panels. "Detach" pops a session into a free-floating, absolutely-positioned element in the same page (not an OS window), so the hand keeps control forever.
• C — One window spanning both monitors. Run Codeman in a single window stretched across both physical displays, so "drag across monitors" is just "drag across the page," and the second monitor's pixels are actually used.

Option B (one OS window per monitor + a distributed BroadcastChannel cursor protocol + cross-window session hand-off) was considered and deferred: it's the only path to independent per-monitor OS windows, but it's a much larger build and reintroduces the cross-window wall this design exists to avoid.

---

Part A — In-page floating panels

Codeman already has the primitive

panels-ui.js has a .detached "floating window": an absolutely-positioned <div> with panel.style.top/left/width/height and a drag handler (setupMonitorDrag) — all in-page, no window.open. The session-tab detach simply picked the wrong primitive (the OS-window one). Part A reuses the in-page one.

Concurrent session rendering — RESOLVED (2026-06-08): live floats are feasible now

The main dashboard view renders one active session at a time (a single shared this.terminal opened into #terminalContainer in terminal-ui.js:26, swapped by selectSession()). But the page is not limited to one terminal — the PR #103 branch already ships three independent in-page concurrent floating-content subsystems we can reuse, so a live floating panel per session needs no new Codeman rendering architecture:

1. Teammate terminals (panels-ui.js teammateTerminals Map + subagent-windows.js). A Map of concurrent live new Terminal() instances, each terminal.open(body)'d into a floating panel, bound to a sessionId + tmux paneTarget, seeded via REST buffer fetch (/api/sessions/:id/teammate-pane-buffer/:pane), input via REST (/api/sessions/:id/teammate-pane-input), with lazy-mount (_lazyPaneTarget) and dispose() cleanup. This is option 2 already built and shipping.
2. Log-viewer windows (panels-ui.js:2632). Concurrent draggable floating windows, each with its own EventSource SSE stream and lifecycle map — proof of the generic "floating window + independent per-window stream" pattern.
3. /session/:id solo route (server.ts:573, renderIndexHtml(soloId), text/event-stream at :633). A full standalone live session page — iframe-able — reusing the exact multi-client fan-out detach already depends on. Solo mode deliberately omits the gesture overlay (server.ts:1010-1014), so there's no nested-overlay problem.

The PTY multi-viewer fan-out is confirmed (the brief's open "yes"): a session is addressable by multiple concurrent clients — the solo route, the teammate REST endpoints, and the on-tab pop-out all view the same live session.

Resolution: skip the static-preview fallback. Build live floats directly, ranked by new-code cost:

• Primary — iframe the solo route. floatPanel(id) = <div class="cg-float" data-id><iframe src="/session/:id"></iframe></div>. The iframe is a complete live session view (its own terminal + stream client); the gesture layer moves the div and the iframe rides along. Lowest new code; reuses proven fan-out; no terminal wiring. Trade-off: a full app shell per float (heavier — fine for a few, watch memory at many).
• Richer alt — native teammate-style panel. Mount a new Terminal() in the float body, seed via the session buffer endpoint, feed via the teammate stream. Native (no iframe), lighter per-float, same-document. Use if the iframe feels heavy or you want tighter integration.

Either way the hand only places the float; typing into it uses a keyboard (focus the iframe / native terminal) — consistent with "gesture places, keyboard types."

Gesture grammar (replaces the current detach path in entry.ts)

The grab/drag/drop plumbing already exists; only the drop action changes.

• Grab a .session-tab[data-id] (unchanged: onGrab, ghost-follow).
• Pull past DETACH_PULL_PX to arm (unchanged: grab.armed).
• Drop while armed → no longer window.app.detachSession. Instead spawn an in-page floating panel for that session id at the drop point. New method floatPanel(id, x, y) replacing detach(id).
• Re-grab a floating panel (new PANEL_SELECTOR, e.g. .cg-float[data-id]) → move it; drop anywhere → it stays. This is the capability detach lost.
• Drop a panel back over the tab strip (or a dock zone) → re-dock (remove the float; session returns to a plain tab). Mirrors the .detached → attach toggle Codeman already has.
• Keep window.open detach as a separate, deliberate verb — e.g. a button, or a distinct "throw up and off-screen" gesture — for intentionally parking a session in its own OS window. It is not the default pinch action anymore.

entry.ts change surface

• New state: floats: Map<string, FloatingPanel> (id → element + position), parallel to the existing grabs/taps maps.
• onGrab: extend hit-testing to also match PANEL_SELECTOR, so an existing float can be re-grabbed (priority: panel over tab when overlapping).
• onDrop: replace if (grab.armed) this.detach(grab.id) with this.floatPanel(...); add the panel re-dock branch.
• floatPanel(id, x, y): create/show the in-page panel (Part A option 1/2), position it absolutely at the drop point. Idempotent per id (re-grab moves the existing one, never duplicates).
• Coordinate mapping is already viewport-pixel based (the click-through surface maps cursor → viewport px), so it needs no change for spanning — see Part C.

---

Part C — One window spanning both monitors

Once the Codeman window physically covers both displays, Part A's panels drag across the seam for free, because the gesture cursor is already in viewport pixels and the viewport now spans both monitors.

macOS setup (operational, near-zero code)

1. System Settings → Desktop & Dock → uncheck "Displays have separate Spaces." (Requires a logout/login.) This is what lets a single window straddle two physical displays.
2. Run Codeman maximized, not fullscreen. Browser fullscreen is per display and will not span — use a maximized/borderless window dragged to cover both monitors. (A kiosk/--app Chrome window sized to the union rect is the cleanest.) Automated by scripts/span-codeman.sh — it reads the display-union rect (Finder desktop bounds) and launches a Brave-first Chromium-family --app window sized to it (fresh per-browser profile so the geometry flags are honored; BROWSER= overrides — plain Chrome bounced on the desk machine). One-click from the Codeman header "multi-monitor" button (POST /api/system/span-displays, which spawns this script), or run it directly. Step 1 + logout is still manual; the script warns if spanning isn't active.
3. Arrange the two monitors as a contiguous rectangle in Display settings so the union has no vertical offset gap.

Coordinate model & the bezel seam

• Enumerate displays with window.getScreenDetails() (Chrome, secure context, window-management permission). Gives each screen's left/top/width/height/availLeft/... in a virtual-desktop coordinate space spanning all monitors.
• Use it for screen-edge snapping zones: e.g. dropping a panel within the right screen's bounds snaps it to fill that screen; the seam between the two screens' rects is a "halt / boundary" zone the cursor crosses.
• Account for the bezel gap. The two monitors are physically separated, but the spanned window's pixels are contiguous — a panel dragged across the seam visually jumps the bezel. Optional: add a dead-band at the seam x-coordinate so a panel snaps to one side rather than straddling.
• getScreenDetails is only needed for snapping/zone logic, not for basic dragging — dragging works the moment the window spans. So Part C can ship in two steps: (1) just span + free drag, (2) add getScreenDetails snapping.

Constraints to surface to the user

• macOS-specific; the "separate Spaces" toggle is global and affects all apps.
• Real fullscreen is unavailable (must run maximized).
• A bezel-width discontinuity sits in the middle of the coordinate space.
• window-management permission prompts once.

---

Build phases

1. ✅ A-MVP — in-page live float, single monitor (DONE, 581fcf9/3e0447a). entry.ts floatSession(id, x, y) spawns an iframe of /session/:id in a .cg-float div (640×420) at the drop point; re-grab + move work (re-dock zone still TBD). Movability restored — the live session rides inside the page. (The method is named floatSession, not the design-sketch floatPanel.)
2. ✅ C-span — span the window (DONE, 063fd8f/59946b8; validated at desk). macOS "separate Spaces" off + scripts/span-codeman.sh launches a maximized --app window across the display union (Brave-first; BROWSER= override). Confirmed: panels drag across the seam — viewport mapping held with zero entry.ts change. Launchable one-click from the Codeman header button → POST /api/system/span-displays (PR #103 95b0035), or by running the script.
3. ⏳ C-snap — screen-aware snapping (pending). Add getScreenDetails; snap a panel to the screen it's dropped on; add the seam dead-band.
4. A-alt (optional). Swap the iframe float for a native teammate-style new Terminal() panel if the iframe shell feels heavy or many floats strain memory.

Open questions / verification before coding

• Concurrent rendering — RESOLVED (2026-06-08). A live session view can be mounted into an arbitrary in-page container concurrently with the main view. The PR #103 branch already ships it three ways (teammate terminals, log-viewer SSE windows, the iframe-able /session/:id solo route); fan-out confirmed. Primary path: iframe the solo route. See the resolved section above.
• Native-panel live feed (only if taking A-alt, not the iframe): trace the exact transport that pushes continuous teammate-pane output after the REST buffer seed (pendingData flush source). The iframe path sidesteps this.
• window.app.detachSession location: when wiring the kept OS-window detach verb, note this method was not found in PR #103 source (only server-side detachSessionListeners); it works at runtime, so confirm where it's actually defined before extending it.
• Spanning feasibility on the actual desk setup (monitor arrangement, whether "separate Spaces" off is acceptable to the user globally).
• Re-dock target: decide the re-dock gesture/zone (drop over tab strip vs a dedicated dock region).

Touch-points summary

Layer	File	Change	Status
Gesture consumer	gesture-proto/src/codeman/entry.ts	detach() → floatSession(); floats map; panel re-grab; (later) re-dock branch + getScreenDetails snapping	✅ float+re-grab done (581fcf9/3e0447a); re-dock/snap pending
Gesture core	gesture-proto/src/gesture/*	none — stays transport-agnostic	✅ unchanged
Codeman — launch	src/web/routes/system-routes.ts, src/web/public/{index.html,panels-ui.js}, scripts/span-codeman.sh	header button → POST /api/system/span-displays → spawn span script (bundled into repo)	✅ PR #103 95b0035
Codeman — caching	src/web/server.ts	cacheBustAssets() — ?v=<mtime> on all same-origin .js/.css so frontend edits show on normal reload	✅ PR #103 b5ea711
Ops	macOS display settings + scripts/span-codeman.sh	"separate Spaces" off + re-login; Brave-first maximized spanning window	✅ validated at desk