# Voice

URL: https://cogitator.app/docs/voice

> Add voice and realtime audio capabilities to Cogitator agents with @cogitator-ai/voice — speech-to-text, text-to-speech, voice activity detection, and native speech-to-speech via OpenAI Realtime and Gemini Live APIs.

## What is @cogitator-ai/voice?

`@cogitator-ai/voice` brings voice capabilities to any Cogitator agent. It supports two fundamentally different approaches to voice interaction:

**Pipeline mode** routes audio through a classic STT -> Agent -> TTS loop. Your agent receives text, responds with text, and the voice package handles all audio conversion. This works with any LLM backend — OpenAI, Anthropic, Google, Ollama, or anything else Cogitator supports.

**Realtime mode** uses native speech-to-speech APIs (OpenAI Realtime or Gemini Live) where the model directly processes and generates audio. Lower latency, more natural prosody, but limited to providers that offer realtime APIs.

```
Pipeline Mode:
┌──────┐    ┌─────┐    ┌───────┐    ┌─────┐    ┌──────┐
│ Mic  │───>│ STT │───>│ Agent │───>│ TTS │───>│ Spkr │
└──────┘    └─────┘    └───────┘    └─────┘    └──────┘

Realtime Mode:
┌──────┐    ┌───────────────────┐    ┌──────┐
│ Mic  │───>│ Realtime LLM API  │───>│ Spkr │
└──────┘    └───────────────────┘    └──────┘
```

## Installation

```bash
pnpm add @cogitator-ai/voice
pnpm add openai             # only for OpenAISTT / OpenAITTS
pnpm add onnxruntime-node   # only for Silero VAD
```

`openai` is an optional peer dependency, needed only for `OpenAISTT` and `OpenAITTS`. They load it on first use, so the package works without it when you use Deepgram, ElevenLabs or the realtime providers; if it is missing, the first `transcribe` / `synthesize` call throws with an install hint. `@cogitator-ai/core` is needed for `createCogitatorRunner()` with a Cogitator agent.

## Quick Start — Pipeline

```typescript
import { Cogitator, Agent } from '@cogitator-ai/core';
import {
  VoiceAgent,
  OpenAISTT,
  OpenAITTS,
  EnergyVAD,
  createCogitatorRunner,
} from '@cogitator-ai/voice';

const cogitator = new Cogitator({ memory: { adapter: 'memory' } });
const agent = new Agent({
  name: 'assistant',
  model: 'openai/gpt-6-luna',
  instructions: 'You are a helpful voice assistant. Keep answers short.',
});

const voiceAgent = new VoiceAgent({
  mode: 'pipeline',
  agent: createCogitatorRunner(cogitator, agent),
  stt: new OpenAISTT({ apiKey: process.env.OPENAI_API_KEY! }),
  tts: new OpenAITTS({ apiKey: process.env.OPENAI_API_KEY! }),
  vad: new EnergyVAD(),
});

await voiceAgent.listen(8080);
```

Connect from any WebSocket client at `ws://localhost:8080/voice` — send binary PCM16 audio frames, receive audio and JSON events back (see [WebSocket protocol](/docs/voice/pipeline#websocket-protocol)).

`createCogitatorRunner()` adapts a `Cogitator` runtime and `Agent` to the `VoiceAgentRunner` interface (`run(input, { sessionId, signal })`): each voice session gets its own memory thread (`voice:<sessionId>`) and interrupted turns abort the run. Pass `{ threadId: (ctx) => ... }` as the third argument to choose threads yourself, e.g. to continue a user's chat history:

```typescript
const runner = createCogitatorRunner(cogitator, agent, {
  threadId: ({ sessionId }) => `call:${sessionId}`,
});
```

Any object with `run(input, context?) => Promise<{ content }>` (and optionally `instructions`) works as the runner.

## Quick Start — Realtime

```typescript
import { VoiceAgent, createCogitatorRunner } from '@cogitator-ai/voice';

const voiceAgent = new VoiceAgent({
  agent: createCogitatorRunner(cogitator, agent),
  mode: 'realtime',
  realtimeProvider: 'gemini',
  realtimeApiKey: process.env.GOOGLE_API_KEY!,
  voice: 'Puck',
});

await voiceAgent.listen(8080);
```

## Components

| Component                                  | Description                                             |
| ------------------------------------------ | ------------------------------------------------------- |
| [STT Providers](/docs/voice/stt-providers) | OpenAI and Deepgram speech-to-text                      |
| [TTS Providers](/docs/voice/tts-providers) | OpenAI and ElevenLabs text-to-speech                    |
| [VAD](/docs/voice/vad)                     | Energy-based and Silero neural voice activity detection |
| [Pipeline](/docs/voice/pipeline)           | Batch and streaming STT -> Agent -> TTS processing      |
| [Realtime](/docs/voice/realtime)           | Native speech-to-speech with OpenAI and Gemini          |

## Audio Format

All raw audio in the voice package is **PCM16**: 16-bit signed integers, little-endian, mono.

| Path                        | Input                                        | Output                                 |
| --------------------------- | -------------------------------------------- | -------------------------------------- |
| Pipeline (STT streams, VAD) | 16kHz (Deepgram and VAD take a `sampleRate`) | MP3 by default, or `ttsOptions.format` |
| OpenAI Realtime             | 24kHz                                        | 24kHz                                  |
| Gemini Live                 | 16kHz                                        | 24kHz                                  |

STT providers also accept containerized files (wav, mp3, ogg, flac, webm, mp4); the container is detected from magic bytes. The pipeline passes its `ttsOptions` to every TTS call; without them it gets the provider's default format (MP3). Set `ttsOptions: { format: 'pcm16' }` on the pipeline or `VoiceAgent` for raw 24kHz PCM (see [TTS options](/docs/voice/pipeline#tts-options)).

The package includes audio conversion utilities:

```typescript
import {
  float32ToPcm16,
  pcm16ToFloat32,
  pcmToWav,
  wavToPcm,
  resample,
  calculateRMS,
  detectAudioFormat,
} from '@cogitator-ai/voice';
```

## Voice Tools

`voiceTools({ stt, tts })` returns two tools for text agents that handle audio as data: `transcribe_audio` (`{ audioBase64, language? }` → `{ text, language, duration }`) and `speak_text` (`{ text, voice?, format? }` → `{ audioBase64, format }`). They are plain `{ name, description, parameters, execute }` objects with Zod parameters — wrap them with `tool()` from `@cogitator-ai/core`:

```typescript
import { Agent, tool } from '@cogitator-ai/core';
import { voiceTools, DeepgramSTT, ElevenLabsTTS } from '@cogitator-ai/voice';

const [transcribe, speak] = voiceTools({
  stt: new DeepgramSTT({ apiKey: process.env.DEEPGRAM_API_KEY! }),
  tts: new ElevenLabsTTS({ apiKey: process.env.ELEVENLABS_API_KEY! }),
});

const agent = new Agent({
  name: 'media-assistant',
  model: 'openai/gpt-5.5',
  instructions: 'Transcribe audio the user sends and answer with speech when asked.',
  tools: [tool(transcribe), tool(speak)],
});
```

`transcribeTool(stt)` and `speakTool(tts)` create them one at a time.

## Next Steps

* [STT Providers](/docs/voice/stt-providers) — configure speech recognition
* [TTS Providers](/docs/voice/tts-providers) — configure speech synthesis
* [VAD](/docs/voice/vad) — detect when users start and stop speaking
* [Pipeline Mode](/docs/voice/pipeline) — batch and streaming voice processing
* [Realtime Mode](/docs/voice/realtime) — native speech-to-speech APIs