Cogitator
Voice

STT Providers

Speech-to-text providers for @cogitator-ai/voice — OpenAI (gpt-transcribe) for batch transcription and Deepgram (nova-3) for real-time streaming with interim results.

Overview

STT providers convert audio to text. Every provider implements the STTProvider interface:

interface STTProvider {
  readonly name: string;
  transcribe(audio: Buffer, options?: STTOptions): Promise<TranscribeResult>;
  createStream(options?: STTStreamOptions): STTStream;
}

Both providers support batch transcription via transcribe() and streaming via createStream(). The key difference is that Deepgram provides true real-time streaming over WebSocket, while OpenAI buffers audio and transcribes on close().

Provider Comparison

FeatureOpenAI STTDeepgram STT
Default modelgpt-transcribenova-3
StreamingBuffered (transcribes on close)Real-time WebSocket
Interim resultsNoYes
Word timestampswhisper-1 onlyYes
EndpointingNoConfigurable
Auto-punctuationYesYes
Other modelsgpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1nova-2, enhanced, base

OpenAI STT

Uses the OpenAI Audio Transcriptions API. Default model is gpt-transcribe.

Requires the optional openai peer (pnpm add openai). It is loaded on the first transcribe call (or the first stream close()), not when you import the package or construct OpenAISTT; if it is missing, that call throws with an install hint.

import { OpenAISTT } from '@cogitator-ai/voice';

const stt = new OpenAISTT({
  apiKey: process.env.OPENAI_API_KEY!,
  model: 'gpt-transcribe',
});

Batch Transcription

const result = await stt.transcribe(audioBuffer, { language: 'en' });
console.log(result.text);

const whisper = new OpenAISTT({ apiKey: process.env.OPENAI_API_KEY!, model: 'whisper-1' });
const detailed = await whisper.transcribe(audioBuffer);
console.log(detailed.duration, detailed.words);
// [{ word: 'Hello', start: 0.0, end: 0.32, confidence: 1 }, ...]

gpt-transcribe and gpt-4o(-mini)-transcribe only support the json response format, so word timestamps and duration are returned for whisper-1 only. For gpt-transcribe, language is sent as a languages list and the language the API detects is returned in result.language. Input can be a containerized file (wav, mp3, ogg, flac, webm, mp4 — detected from magic bytes) or headerless PCM16 mono 16kHz, which is wrapped into WAV automatically.

Streaming

OpenAI STT streaming collects raw PCM16 chunks (at sampleRate, default 16kHz) and transcribes the complete buffer as WAV when close() is called. Calling close() twice returns the same result; empty streams do not hit the API.

const stream = stt.createStream({ language: 'en' });

stream.on('final', (result) => {
  console.log('Transcription:', result.text);
});

stream.write(audioChunk1);
stream.write(audioChunk2);
const result = await stream.close();

Configuration

FieldTypeDefaultDescription
apiKeystringrequiredOpenAI API key
modelstringgpt-transcribeModel ID
baseURLstring—Custom API base URL

Deepgram STT

Uses the Deepgram API with real-time WebSocket streaming. Default model is nova-3.

import { DeepgramSTT } from '@cogitator-ai/voice';

const stt = new DeepgramSTT({
  apiKey: process.env.DEEPGRAM_API_KEY!,
  model: 'nova-3',
  language: 'en',
});

Batch Transcription

const result = await stt.transcribe(audioBuffer);

console.log(result.text);
console.log(result.duration);
console.log(result.words);
// [{ word: 'Hello', start: 0.0, end: 0.32, confidence: 0.99 }, ...]

Files with a container (wav, mp3, ogg, flac, webm, mp4) are sent as-is; headerless input is sent as raw PCM16 mono at sampleRate. Requests time out after 30 seconds.

Streaming

Deepgram streaming opens a WebSocket connection and provides real-time results as audio arrives. Interim results (on by default) update continuously; a final event is emitted for every finalized segment, and close() resolves with all final segments joined, their words and the total duration. Audio written before the socket opens is queued. close() waits up to 10 seconds for Deepgram to flush; if the connection drops before close(), an error is emitted.

const stream = stt.createStream({
  interimResults: true,
  endpointing: 500,
  language: 'en',
});

stream.on('partial', (text) => {
  console.log('Interim:', text);
});

stream.on('final', (result) => {
  console.log('Final:', result.text);
});

stream.on('error', (err) => {
  console.error('Stream error:', err);
});

stream.write(audioChunk1);
stream.write(audioChunk2);
await stream.close();

Configuration

FieldTypeDefaultDescription
apiKeystringrequiredDeepgram API key
modelstringnova-3Model ID
languagestring—Default language code
sampleRatenumber16000Sample rate of raw PCM16 (streams and headerless batch input)

STT Options

Options for transcribe() (and createStream()):

FieldTypeDescription
languagestringLanguage code (e.g. "en", "es", "ja")
promptstringContext hint to improve transcription accuracy (OpenAI only)

Additional options for createStream():

FieldTypeDescription
interimResultsbooleanPartial/interim results (Deepgram, default true)
endpointingnumberSilence duration (ms) for Deepgram endpointing
sampleRatenumberSample rate of the raw PCM16 written to the stream (default 16000)

TranscribeResult

Both providers return the same result shape:

interface TranscribeResult {
  text: string;
  language?: string;
  duration?: number;
  words?: Array<{
    word: string;
    start: number;
    end: number;
    confidence: number;
  }>;
}

Custom STT Provider

Implement STTProvider to integrate any speech recognition service:

import { EventEmitter } from 'node:events';
import type {
  STTProvider,
  STTOptions,
  STTStream,
  STTStreamOptions,
  TranscribeResult,
} from '@cogitator-ai/voice';

class BufferedStream extends EventEmitter implements STTStream {
  private chunks: Buffer[] = [];

  constructor(
    private readonly stt: STTProvider,
    private readonly options?: STTStreamOptions
  ) {
    super();
  }

  write(chunk: Buffer): void {
    this.chunks.push(chunk);
  }

  async close(): Promise<TranscribeResult> {
    const result = await this.stt.transcribe(Buffer.concat(this.chunks), this.options);
    this.emit('final', result);
    return result;
  }
}

class WhisperLocalSTT implements STTProvider {
  readonly name = 'whisper-local';

  async transcribe(audio: Buffer, options?: STTOptions): Promise<TranscribeResult> {
    // call your local Whisper instance
    return { text: 'transcribed text' };
  }

  createStream(options?: STTStreamOptions): STTStream {
    return new BufferedStream(this, options);
  }
}

The pipeline writes raw PCM16 (16kHz) chunks to the stream, listens for partial / final / error, and uses the result of close() as the user's turn.

Voice

Add voice and realtime audio capabilities to Cogitator agents with @cogitator-ai/voice — speech-to-text, text-to-speech, voice activity detection, and native speech-to-speech via OpenAI Realtime and Gemini Live APIs.

TTS Providers

Text-to-speech providers for @cogitator-ai/voice — OpenAI (gpt-4o-mini-tts with 13 voices and instructions) and ElevenLabs (eleven_flash_v2_5 with low-latency streaming).

On this page

OverviewProvider ComparisonOpenAI STTBatch TranscriptionStreamingConfigurationDeepgram STTBatch TranscriptionStreamingConfigurationSTT OptionsTranscribeResultCustom STT Provider