Realtime Mode

OverviewCopy Anchor Link

Realtime mode bypasses the STT -> Agent -> TTS pipeline entirely. Instead, the LLM directly processes incoming audio and generates audio responses. This results in lower latency and more natural prosody, since the model controls the voice output directly.

RealtimeSession is a unified interface over two provider-specific adapters:

• OpenAI Realtime API (GA) — WebSocket connection to gpt-realtime-2.1-mini with server-side VAD, input transcription and tool calling
• Gemini Live API — WebSocket connection to gemini-3.8-live with native audio, input/output transcription and tool calling

RealtimeSessionCopy Anchor Link

The main class for realtime voice interaction. It wraps the provider-specific adapter and exposes a consistent event-based API.

import { RealtimeSession } from '@cogitator-ai/voice';

const session = new RealtimeSession({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY!,
  model: 'gpt-realtime-2.1-mini',
  voice: 'marin',
  instructions: 'You are a helpful assistant.',
});

ConnectingCopy Anchor Link

await session.connect();

Opens a WebSocket connection to the provider. For OpenAI, this sends a GA session.update (type: 'realtime', audio formats, voice, transcription, instructions, tools). For Gemini, this sends a setup message with model config, transcription and tools and resolves on setupComplete. A failed connection rejects without emitting error.

Audio and text sent before connect() resolves are queued (audio capped at 1MB) and flushed in order once the session is ready.

Sending AudioCopy Anchor Link

session.pushAudio(pcm16Chunk);

Send PCM16 mono frames — 24kHz for OpenAI, 16kHz for Gemini. Both providers handle VAD server-side — they detect when the user starts and stops speaking. Model audio is always PCM16 24kHz.

Sending TextCopy Anchor Link

session.sendText('Hello!');

Send a text message instead of audio. The model responds with audio.

InterruptionCopy Anchor Link

session.interrupt();

Cancel the current response. For OpenAI, sends response.cancel (only while a response is active). Gemini barge-in is driven by user audio, so interrupt() drops the remaining model audio of the current turn locally; when Gemini detects the user talking over the model it reports interrupted and the session emits speech_start.

ClosingCopy Anchor Link

session.close();

Closes the WebSocket connection and drops queued input; a closed session cannot reconnect. session.provider and session.isConnected report its state. The provider adapters (OpenAIRealtimeAdapter, GeminiRealtimeAdapter) are exported for direct use.

EventsCopy Anchor Link

session.on('connected', () => {
  console.log('WebSocket connected');
});

session.on('audio', (chunk: Buffer) => {
  playAudio(chunk);
});

session.on('transcript', (text: string, role: 'user' | 'assistant') => {
  console.log(`${role}: ${text}`);
});

session.on('speech_start', () => {
  console.log('Server VAD: speech detected');
});

session.on('tool_call', (name: string, args: unknown) => {
  console.log(`Tool called: ${name}`, args);
});

session.on('error', (error: Error) => {
  console.error('Realtime error:', error);
});

OpenAI Realtime APICopy Anchor Link

Default model: gpt-realtime-2.1-mini (also gpt-realtime-2.1). Connects via WebSocket to wss://api.openai.com/v1/realtime using the GA interface (the beta OpenAI-Beta: realtime=v1 interface was shut down in May 2026).

const session = new RealtimeSession({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY!,
  model: 'gpt-realtime-2.1-mini',
  voice: 'marin',
  instructions: 'You are a friendly assistant who speaks concisely.',
  tools: [
    {
      name: 'get_weather',
      description: 'Get current weather for a city',
      parameters: {
        type: 'object',
        properties: { city: { type: 'string' } },
        required: ['city'],
      },
      execute: async (args) => {
        const { city } = args as { city: string };
        return { temperature: 72, unit: 'F', city };
      },
    },
  ],
});

await session.connect();
session.pushAudio(pcm16Chunk);

OpenAI FeaturesCopy Anchor Link

• Server-side VAD — OpenAI handles turn detection, emits speech_start events
• Tool calling — all function calls of a completed response are executed, their outputs sent back, then a single follow-up response is requested; unknown tools and invalid arguments are answered with an error output
• Transcription — user speech transcripts (gpt-live-transcribe) and the model's audio transcripts via the transcript event
• Audio format — PCM16 24kHz input and output

OpenAI VoicesCopy Anchor Link

alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin (default), cedar

Gemini Live APICopy Anchor Link

Default model: gemini-3.8-live (also gemini-3.1-flash-live-preview). Connects via WebSocket to the Gemini Developer API BidiGenerateContent endpoint, authenticating with the x-goog-api-key header.

const session = new RealtimeSession({
  provider: 'gemini',
  apiKey: process.env.GOOGLE_API_KEY!,
  model: 'gemini-3.8-live',
  voice: 'Puck',
  instructions: 'You are a helpful assistant.',
  tools: [
    {
      name: 'search',
      description: 'Search the web',
      parameters: {
        type: 'object',
        properties: { query: { type: 'string' } },
      },
      execute: async (args) => {
        const { query } = args as { query: string };
        return { results: [`Result for: ${query}`] };
      },
    },
  ],
});

await session.connect();
session.pushAudio(pcm16Chunk);

Gemini FeaturesCopy Anchor Link

• Native audio generation — model generates audio directly
• Transcription — input and output audio transcription is enabled; streamed fragments are emitted as one transcript per turn (user first, then assistant). Model "thought" parts are never emitted
• Tool calling — function declarations sent during setup, structured results ({ result } / { error }) returned via toolResponse
• Audio format — PCM16 16kHz input (realtimeInput.audio), PCM16 24kHz output

Gemini VoicesCopy Anchor Link

Puck, Charon, Kore, Fenrir, Aoede

Provider ComparisonCopy Anchor Link

Realtime with VoiceAgentCopy Anchor Link

Use VoiceAgent to expose a realtime session over WebSocket transport:

import { VoiceAgent, createCogitatorRunner, type RealtimeTool } from '@cogitator-ai/voice';

const weatherTool: RealtimeTool = {
  name: 'get_weather',
  description: 'Get current weather for a city',
  parameters: { type: 'object', properties: { city: { type: 'string' } }, required: ['city'] },
  execute: async (args) => ({ city: (args as { city: string }).city, temperature: 21, unit: 'C' }),
};

const voiceAgent = new VoiceAgent({
  agent: createCogitatorRunner(cogitator, agent),
  mode: 'realtime',
  realtimeProvider: 'openai',
  realtimeApiKey: process.env.OPENAI_API_KEY!,
  realtimeModel: 'gpt-realtime-2.1-mini',
  voice: 'marin',
  tools: [weatherTool],
});

await voiceAgent.listen(8080);

Each WebSocket client gets its own RealtimeSession. The session instructions come from instructions or, if unset, from agent.instructions; tools are executed server-side. The model talks to the user directly — agent.run() is not called in realtime mode, so the Cogitator agent's tools and memory are not used; give the session its own tools. Audio from the client is forwarded to the realtime API, and audio/transcript/speech_start/turn_end events are forwarded back. Clients can send { type: 'text', text } and { type: 'interrupt' } control messages. If the provider connection fails or drops, the client is closed with code 1011.

RealtimeSessionConfigCopy Anchor Link

RealtimeToolCopy Anchor Link

interface RealtimeTool {
  name: string;
  description: string;
  parameters: Record<string, unknown>;
  execute: (args: unknown) => Promise<unknown>;
}