Codeman/docs/claude-voice-plan.md

Claude voice dictation in Codeman

Wire Codeman's existing mic button to the same speech-to-text service Claude Code's own /voice mode uses, so dictation works with no third-party API key for anyone already signed in to Claude Code on the server.

Why the CLI's own voice mode cannot be reused directly

Claude Code 2.1.x ships voice input: /voice hold|tap|off arms it, the CLI opens the host's microphone (native audio-capture-napi, falling back to sox/arecord on Linux after probing /proc/asound/cards), streams PCM upstream and types the transcript into its own composer.

Every part of that is on the wrong machine for Codeman. The CLI runs inside a tmux pane on the server, which is typically headless and has no sound card at all, while the human is in a browser on a phone somewhere else. Toggling /voice in the pane from Codeman would arm a microphone nobody is sitting in front of. So Codeman keeps capturing audio in the browser, where the user actually is, and only borrows the CLI's transcription backend.

The backend, as the CLI uses it

Extracted from the 2.1.226 binary (connectVoiceStream):

URL	wss://api.anthropic.com/api/ws/speech_to_text/voice_stream
Query	encoding=linear16, sample_rate=16000, channels=1, endpointing_ms=300, utterance_end_ms=1000, language=<lang>, use_conversation_engine=true, stt_provider=deepgram-nova3
Headers	Authorization: Bearer <Claude Code OAuth access token>, User-Agent, x-app: cli, anthropic-client-platform, optional x-config-keyterms
Audio	raw binary frames, PCM signed 16-bit little-endian, 16 kHz, mono
Keepalive	{"type":"KeepAlive"} on open, then every 8 s
Finalize	{"type":"CloseStream"}, then wait for the endpoint frame
Downstream	{"type":"TranscriptText"|"TranscriptInterim","data":"…"} (running interim), {"type":"TranscriptEndpoint"} (promotes the pending interim to final), {"type":"TranscriptError",…}, {"type":"error","message":…}

Deepgram Nova-3 runs server-side, so the Deepgram-quality result arrives without a Deepgram account. Verified against the live endpoint before this design was written: connect, stream PCM, receive interims and an endpoint frame.

Architecture

The browser cannot call that endpoint itself: it would need the OAuth bearer token in page JavaScript (and CORS would refuse anyway). So the audio goes browser → Codeman → Anthropic, and Codeman is the only thing that ever touches the token.

mic → AudioWorklet (Float32 → PCM16 @16 kHz)
    → wss://<codeman>/ws/voice/stream          [cookie/basic auth, Origin+Host guarded]
    → VoiceStreamRelay (reads ~/.claude/.credentials.json per connect)
    → wss://api.anthropic.com/api/ws/speech_to_text/voice_stream
    ← {"t":"transcript","text":…,"final":…}    → existing _insertText() path

Nothing about the insert path changes: the transcript lands in the same preview overlay, the same direct/compose insert modes, the same green Send button.

Server pieces

• src/claude-credentials.ts — locate and parse the Claude Code OAuth credentials. parseClaudeCredentials() is pure (JSON string + now → status) and unit-tested; readClaudeOAuthToken() wraps it with IO: $CLAUDE_CONFIG_DIR/.credentials.json or ~/.claude/.credentials.json, and on macOS the login keychain (security find-generic-password -s "Claude Code-credentials"). Read-only, always. Codeman never writes credentials and never refreshes the token: a refresh rotates the refresh token, and racing Claude Code's own refresh could sign the user out of their CLI. An expired token surfaces as a plain "run a Claude session to refresh" error instead. The token is never logged, never returned by any endpoint, and never sent to the browser.

• src/web/voice-stream.ts — pure buildVoiceStreamUrl() / buildVoiceStreamHeaders() / sanitizeKeyterms() (ASCII-only, deduped, 1024-char cap, mirroring the CLI), plus VoiceStreamRelay, which owns one upstream socket: keepalive timer, audio passthrough, transcript translation, finalize, and the caps below.

• src/web/routes/voice-routes.ts

  • GET /api/voice/status → { available, reason, subscriptionType?, expiresAt? }. Never the token. available:false with a machine-readable reason (disabled, no-credentials, expired) is what the settings row and the provider resolver read.
  • GET /ws/voice/stream?language=&keyterms= → the relay. Same upgrade guard as /ws/sessions/:id/terminal: allowed Host, same-site Origin, and the global auth hook has already run on the handshake.

Caps, because an open mic is an open pipe: one stream per connection, MAX_VOICE_STREAMS concurrent server-wide, a hard MAX_STREAM_MS per stream, and a per-frame size cap. A tab left recording cannot bill an unbounded amount of upstream audio.

Frontend pieces

• voice-pcm-worklet.js — an AudioWorkletProcessor converting Float32 blocks to PCM16 and posting ~256 ms frames back. MediaRecorder cannot produce raw PCM, which is why the existing Deepgram path (container audio, auto-detected) cannot be reused as-is. Falls back to ScriptProcessorNode where AudioWorklet is unavailable.
• ClaudeVoiceProvider in voice-input.js — mirrors DeepgramProvider's shape (start({language, keyterms, onStream, onResult, onError, onEnd})) so VoiceInput treats the three providers uniformly.
• Provider resolution — new voiceSettings.provider: auto (default) | claude | deepgram | webspeech. auto picks Claude when /api/voice/status reports it available, else Deepgram when a key is set, else Web Speech. Pinning a provider always wins, so an existing Deepgram user can keep exactly what they have.

Settings

• claudeVoiceEnabled — synced, default OFF, gating the whole server side. Off is the honest default: turning it on means this machine's Claude subscription starts paying for transcription for whoever can reach the UI, and the audio goes to Anthropic rather than to wherever it went before. One switch in Settings → Voice, and the mic works with no key.
• voiceSettings.provider — per the resolution table above; joins the existing synced voiceSettings object.

Things worth knowing

• This uses an undocumented endpoint with subscription credentials. It is the user's own token, on the user's own machine, driving the user's own Claude Code install, but it is not a published API and Anthropic can change or restrict it. Default-OFF is deliberate; the Deepgram and Web Speech paths stay untouched as the supported fallbacks.
• Multi-user mode: every user's dictation would run on the server owner's Claude credentials, exactly as every user's sessions already run on them. Consistent, but worth stating out loud in the settings copy.
• Token lifetime is about 8 hours, refreshed by Claude Code itself whenever it runs. The relay re-reads the file on every connect rather than caching, so a refresh is picked up on the next press of the mic.
• HTTPS or localhost: getUserMedia needs a secure context. Prod is HTTPS behind tailscale serve, so this is already satisfied; the existing error copy covers the rest.