# Realtime Mode

URL: https://cogitator.app/docs/voice/realtime

> Native speech-to-speech with RealtimeSession — connect to OpenAI Realtime API or Gemini Live API for low-latency voice conversations with tool calling support.

## Overview

Realtime mode bypasses the STT -> Agent -> TTS pipeline entirely. Instead, the LLM directly processes incoming audio and generates audio responses. This results in lower latency and more natural prosody, since the model controls the voice output directly.

`RealtimeSession` is a unified interface over two provider-specific adapters:

* **OpenAI Realtime API (GA)** — WebSocket connection to `gpt-realtime-2.1-mini` with server-side VAD, input transcription and tool calling
* **Gemini Live API** — WebSocket connection to `gemini-3.8-live` with native audio, input/output transcription and tool calling

## RealtimeSession

The main class for realtime voice interaction. It wraps the provider-specific adapter and exposes a consistent event-based API.

```typescript
import { RealtimeSession } from '@cogitator-ai/voice';

const session = new RealtimeSession({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY!,
  model: 'gpt-realtime-2.1-mini',
  voice: 'marin',
  instructions: 'You are a helpful assistant.',
});
```

### Connecting

```typescript
await session.connect();
```

Opens a WebSocket connection to the provider. For OpenAI, this sends a GA `session.update` (`type: 'realtime'`, audio formats, voice, transcription, instructions, tools). For Gemini, this sends a setup message with model config, transcription and tools and resolves on `setupComplete`. A failed connection rejects without emitting `error`.

Audio and text sent before `connect()` resolves are queued (audio capped at 1MB) and flushed in order once the session is ready.

### Sending Audio

```typescript
session.pushAudio(pcm16Chunk);
```

Send PCM16 mono frames — **24kHz for OpenAI**, **16kHz for Gemini**. Both providers handle VAD server-side — they detect when the user starts and stops speaking. Model audio is always PCM16 24kHz.

### Sending Text

```typescript
session.sendText('Hello!');
```

Send a text message instead of audio. The model responds with audio.

### Interruption

```typescript
session.interrupt();
```

Cancel the current response. For OpenAI, sends `response.cancel` (only while a response is active). Gemini barge-in is driven by user audio, so `interrupt()` drops the remaining model audio of the current turn locally; when Gemini detects the user talking over the model it reports `interrupted` and the session emits `speech_start`.

### Closing

```typescript
session.close();
```

Closes the WebSocket connection and drops queued input; a closed session cannot reconnect. `session.provider` and `session.isConnected` report its state. The provider adapters (`OpenAIRealtimeAdapter`, `GeminiRealtimeAdapter`) are exported for direct use.

### Events

```typescript
session.on('connected', () => {
  console.log('WebSocket connected');
});

session.on('audio', (chunk: Buffer) => {
  playAudio(chunk);
});

session.on('transcript', (text: string, role: 'user' | 'assistant') => {
  console.log(`${role}: ${text}`);
});

session.on('speech_start', () => {
  console.log('Server VAD: speech detected');
});

session.on('tool_call', (name: string, args: unknown) => {
  console.log(`Tool called: ${name}`, args);
});

session.on('error', (error: Error) => {
  console.error('Realtime error:', error);
});
```

| Event          | Payload           | Description                                                 |
| -------------- | ----------------- | ----------------------------------------------------------- |
| `connected`    | —                 | Session ready (OpenAI socket open / Gemini `setupComplete`) |
| `disconnected` | `(code, reason)`  | Connection closed after it was established                  |
| `audio`        | `(chunk: Buffer)` | Audio response chunk from the model (PCM16 24kHz)           |
| `transcript`   | `(text, role)`    | Full transcription of a user or assistant turn              |
| `speech_start` | —                 | Server-side VAD detected speech / user barged in            |
| `tool_call`    | `(name, args)`    | Model invoked a tool                                        |
| `turn_end`     | —                 | Model finished its turn                                     |
| `error`        | `(error: Error)`  | Connection or processing error                              |

## OpenAI Realtime API

Default model: `gpt-realtime-2.1-mini` (also `gpt-realtime-2.1`). Connects via WebSocket to `wss://api.openai.com/v1/realtime` using the GA interface (the beta `OpenAI-Beta: realtime=v1` interface was shut down in May 2026).

```typescript
const session = new RealtimeSession({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY!,
  model: 'gpt-realtime-2.1-mini',
  voice: 'marin',
  instructions: 'You are a friendly assistant who speaks concisely.',
  tools: [
    {
      name: 'get_weather',
      description: 'Get current weather for a city',
      parameters: {
        type: 'object',
        properties: { city: { type: 'string' } },
        required: ['city'],
      },
      execute: async (args) => {
        const { city } = args as { city: string };
        return { temperature: 72, unit: 'F', city };
      },
    },
  ],
});

await session.connect();
session.pushAudio(pcm16Chunk);
```

### OpenAI Features

* **Server-side VAD** — OpenAI handles turn detection, emits `speech_start` events
* **Tool calling** — all function calls of a completed response are executed, their outputs sent back, then a single follow-up response is requested; unknown tools and invalid arguments are answered with an error output
* **Transcription** — user speech transcripts (`gpt-live-transcribe`) and the model's audio transcripts via the `transcript` event
* **Audio format** — PCM16 24kHz input and output

### OpenAI Voices

`alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, `marin` (default), `cedar`

## Gemini Live API

Default model: `gemini-3.8-live` (also `gemini-3.1-flash-live-preview`). Connects via WebSocket to the Gemini Developer API `BidiGenerateContent` endpoint, authenticating with the `x-goog-api-key` header.

```typescript
const session = new RealtimeSession({
  provider: 'gemini',
  apiKey: process.env.GOOGLE_API_KEY!,
  model: 'gemini-3.8-live',
  voice: 'Puck',
  instructions: 'You are a helpful assistant.',
  tools: [
    {
      name: 'search',
      description: 'Search the web',
      parameters: {
        type: 'object',
        properties: { query: { type: 'string' } },
      },
      execute: async (args) => {
        const { query } = args as { query: string };
        return { results: [`Result for: ${query}`] };
      },
    },
  ],
});

await session.connect();
session.pushAudio(pcm16Chunk);
```

### Gemini Features

* **Native audio generation** — model generates audio directly
* **Transcription** — input and output audio transcription is enabled; streamed fragments are emitted as one `transcript` per turn (user first, then assistant). Model "thought" parts are never emitted
* **Tool calling** — function declarations sent during setup, structured results (`{ result }` / `{ error }`) returned via `toolResponse`
* **Audio format** — PCM16 16kHz input (`realtimeInput.audio`), PCM16 24kHz output

### Gemini Voices

`Puck`, `Charon`, `Kore`, `Fenrir`, `Aoede`

## Provider Comparison

| Feature          | OpenAI Realtime         | Gemini Live                                        |
| ---------------- | ----------------------- | -------------------------------------------------- |
| Default model    | `gpt-realtime-2.1-mini` | `gemini-3.8-live`                                  |
| Server-side VAD  | Yes                     | Yes                                                |
| Tool calling     | Yes                     | Yes                                                |
| User transcripts | Yes                     | Yes                                                |
| Text input       | Yes                     | Yes                                                |
| Interruption     | `response.cancel`       | Local audio drop; server `interrupted` on barge-in |
| Input audio      | PCM16 24kHz             | PCM16 16kHz                                        |
| Output audio     | PCM16 24kHz             | PCM16 24kHz                                        |
| Default voice    | `marin`                 | `Puck`                                             |

## Realtime with VoiceAgent

Use `VoiceAgent` to expose a realtime session over WebSocket transport:

```typescript
import { VoiceAgent, createCogitatorRunner, type RealtimeTool } from '@cogitator-ai/voice';

const weatherTool: RealtimeTool = {
  name: 'get_weather',
  description: 'Get current weather for a city',
  parameters: { type: 'object', properties: { city: { type: 'string' } }, required: ['city'] },
  execute: async (args) => ({ city: (args as { city: string }).city, temperature: 21, unit: 'C' }),
};

const voiceAgent = new VoiceAgent({
  agent: createCogitatorRunner(cogitator, agent),
  mode: 'realtime',
  realtimeProvider: 'openai',
  realtimeApiKey: process.env.OPENAI_API_KEY!,
  realtimeModel: 'gpt-realtime-2.1-mini',
  voice: 'marin',
  tools: [weatherTool],
});

await voiceAgent.listen(8080);
```

Each WebSocket client gets its own `RealtimeSession`. The session instructions come from `instructions` or, if unset, from `agent.instructions`; `tools` are executed server-side. The model talks to the user directly — `agent.run()` is not called in realtime mode, so the Cogitator agent's tools and memory are not used; give the session its own `tools`. Audio from the client is forwarded to the realtime API, and audio/`transcript`/`speech_start`/`turn_end` events are forwarded back. Clients can send `{ type: 'text', text }` and `{ type: 'interrupt' }` control messages. If the provider connection fails or drops, the client is closed with code `1011`.

## RealtimeSessionConfig

| Field          | Type                   | Required | Description                                                 |
| -------------- | ---------------------- | -------- | ----------------------------------------------------------- |
| `provider`     | `'openai' \| 'gemini'` | Yes      | Realtime API provider                                       |
| `apiKey`       | `string`               | Yes      | API key                                                     |
| `model`        | `string`               | No       | Model override                                              |
| `instructions` | `string`               | No       | System instructions                                         |
| `voice`        | `string`               | No       | Voice name (default: `marin` for OpenAI, `Puck` for Gemini) |
| `tools`        | `Array<RealtimeTool>`  | No       | Tools the model can invoke                                  |

### RealtimeTool

```typescript
interface RealtimeTool {
  name: string;
  description: string;
  parameters: Record<string, unknown>;
  execute: (args: unknown) => Promise<unknown>;
}
```