mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-10-09 00:49:41 +02:00
feat(voice): dictate through the server's Claude Code login, no API key
The mic button previously needed a Deepgram API key, or fell back to the browser's Web Speech engine. It can now transcribe through the same speech-to-text service Claude Code's own /voice mode uses, so anyone signed in to Claude Code on the server gets dictation with no third-party account. Claude Code's voice mode cannot be driven directly: it opens the HOST's microphone (sox/arecord), and the CLI runs in a headless tmux pane while the human is in a browser somewhere else. So capture stays in the browser and only the transcription backend is borrowed. Audio goes browser -> Codeman -> Anthropic. The OAuth token never reaches the page: the browser sends PCM16 (16 kHz mono, produced by an AudioWorklet since MediaRecorder cannot emit raw PCM) and receives text. - GET /api/voice/status reports readiness and never the token - GET /ws/voice/stream relays one dictation, with the same Host/Origin upgrade guard as the terminal socket, plus caps on concurrency, stream length and frame size - credentials are read-only: Codeman never refreshes them, since a refresh rotates the refresh token and could sign the user out of their own CLI - claudeVoiceEnabled (synced, default OFF) gates the whole server side - voiceSettings.provider picks auto/claude/deepgram/webspeech; auto prefers Claude, then a configured Deepgram key, then the browser Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,57 @@
|
||||
/**
|
||||
* @fileoverview AudioWorklet that turns microphone audio into the PCM frames the
|
||||
* Claude voice endpoint expects.
|
||||
*
|
||||
* The endpoint is opened as `encoding=linear16, sample_rate=16000, channels=1`,
|
||||
* i.e. raw signed 16-bit little-endian mono. MediaRecorder cannot produce that
|
||||
* (it only emits container formats — webm/opus, mp4), which is why the Deepgram
|
||||
* path's capture code cannot be reused here: Deepgram sniffs the container,
|
||||
* Anthropic's endpoint does not.
|
||||
*
|
||||
* Sample rate is handled by the AudioContext, constructed at 16000 Hz so the
|
||||
* browser resamples the mic for us. This processor only converts Float32 [-1,1]
|
||||
* to Int16 and batches, because a raw 128-sample render quantum is a ~4 ms
|
||||
* WebSocket frame — 250 frames a second of pure overhead.
|
||||
*
|
||||
* Loaded via `audioWorklet.addModule()` from voice-input.js. Runs on the audio
|
||||
* thread: no DOM, no globals from the page.
|
||||
*
|
||||
* ⚠️ Edit this file and voice-input.js together. Static assets are served
|
||||
* `immutable` for a year and this one is fetched from JS, so it inherits its
|
||||
* cache-bust token from voice-input.js's script tag (see `_workletUrl()`); a
|
||||
* change here alone would keep serving the old copy to every returning browser.
|
||||
*/
|
||||
|
||||
/** ~256 ms at 16 kHz. Big enough to keep frame overhead down, small enough that interim transcripts stay live. */
|
||||
const FRAME_SAMPLES = 4096;
|
||||
|
||||
class PcmFrameProcessor extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this._buffer = new Int16Array(FRAME_SAMPLES);
|
||||
this._offset = 0;
|
||||
}
|
||||
|
||||
process(inputs) {
|
||||
const channel = inputs[0]?.[0];
|
||||
// No input yet (mic still warming) — keep the processor alive.
|
||||
if (!channel) return true;
|
||||
|
||||
for (let i = 0; i < channel.length; i++) {
|
||||
// Clamp before scaling: values slightly outside [-1,1] are legal in Web Audio
|
||||
// and would wrap around to the opposite sign as Int16, which sounds like a click.
|
||||
const sample = Math.max(-1, Math.min(1, channel[i]));
|
||||
this._buffer[this._offset++] = sample < 0 ? sample * 0x8000 : sample * 0x7fff;
|
||||
|
||||
if (this._offset === FRAME_SAMPLES) {
|
||||
// Transfer a copy: the worklet keeps reusing its own buffer.
|
||||
const frame = new Int16Array(this._buffer);
|
||||
this.port.postMessage(frame.buffer, [frame.buffer]);
|
||||
this._offset = 0;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor('pcm-frame-processor', PcmFrameProcessor);
|
||||
Reference in New Issue
Block a user