Voice

What is @cogitator-ai/voice?Copy Anchor Link

@cogitator-ai/voice brings voice capabilities to any Cogitator agent. It supports two fundamentally different approaches to voice interaction:

Pipeline mode routes audio through a classic STT -> Agent -> TTS loop. Your agent receives text, responds with text, and the voice package handles all audio conversion. This works with any LLM backend — OpenAI, Anthropic, Google, Ollama, or anything else Cogitator supports.

Realtime mode uses native speech-to-speech APIs (OpenAI Realtime or Gemini Live) where the model directly processes and generates audio. Lower latency, more natural prosody, but limited to providers that offer realtime APIs.

Pipeline Mode:
┌──────┐    ┌─────┐    ┌───────┐    ┌─────┐    ┌──────┐
│ Mic  │───>│ STT │───>│ Agent │───>│ TTS │───>│ Spkr │
└──────┘    └─────┘    └───────┘    └─────┘    └──────┘

Realtime Mode:
┌──────┐    ┌───────────────────┐    ┌──────┐
│ Mic  │───>│ Realtime LLM API  │───>│ Spkr │
└──────┘    └───────────────────┘    └──────┘

InstallationCopy Anchor Link

pnpm add @cogitator-ai/voice
pnpm add openai             # only for OpenAISTT / OpenAITTS
pnpm add onnxruntime-node   # only for Silero VAD

openai is an optional peer dependency, needed only for OpenAISTT and OpenAITTS. They load it on first use, so the package works without it when you use Deepgram, ElevenLabs or the realtime providers; if it is missing, the first transcribe / synthesize call throws with an install hint. @cogitator-ai/core is needed for createCogitatorRunner() with a Cogitator agent.

Quick Start — PipelineCopy Anchor Link

import { Cogitator, Agent } from '@cogitator-ai/core';
import {
  VoiceAgent,
  OpenAISTT,
  OpenAITTS,
  EnergyVAD,
  createCogitatorRunner,
} from '@cogitator-ai/voice';

const cogitator = new Cogitator({ memory: { adapter: 'memory' } });
const agent = new Agent({
  name: 'assistant',
  model: 'openai/gpt-6-luna',
  instructions: 'You are a helpful voice assistant. Keep answers short.',
});

const voiceAgent = new VoiceAgent({
  mode: 'pipeline',
  agent: createCogitatorRunner(cogitator, agent),
  stt: new OpenAISTT({ apiKey: process.env.OPENAI_API_KEY! }),
  tts: new OpenAITTS({ apiKey: process.env.OPENAI_API_KEY! }),
  vad: new EnergyVAD(),
});

await voiceAgent.listen(8080);

Connect from any WebSocket client at ws://localhost:8080/voice — send binary PCM16 audio frames, receive audio and JSON events back (see WebSocket protocol).

createCogitatorRunner() adapts a Cogitator runtime and Agent to the VoiceAgentRunner interface (run(input, { sessionId, signal })): each voice session gets its own memory thread (voice:<sessionId>) and interrupted turns abort the run. Pass { threadId: (ctx) => ... } as the third argument to choose threads yourself, e.g. to continue a user's chat history:

const runner = createCogitatorRunner(cogitator, agent, {
  threadId: ({ sessionId }) => `call:${sessionId}`,
});

Any object with run(input, context?) => Promise<{ content }> (and optionally instructions) works as the runner.

Quick Start — RealtimeCopy Anchor Link

import { VoiceAgent, createCogitatorRunner } from '@cogitator-ai/voice';

const voiceAgent = new VoiceAgent({
  agent: createCogitatorRunner(cogitator, agent),
  mode: 'realtime',
  realtimeProvider: 'gemini',
  realtimeApiKey: process.env.GOOGLE_API_KEY!,
  voice: 'Puck',
});

await voiceAgent.listen(8080);

ComponentsCopy Anchor Link

Audio FormatCopy Anchor Link

All raw audio in the voice package is PCM16: 16-bit signed integers, little-endian, mono.

STT providers also accept containerized files (wav, mp3, ogg, flac, webm, mp4); the container is detected from magic bytes. The pipeline passes its ttsOptions to every TTS call; without them it gets the provider's default format (MP3). Set ttsOptions: { format: 'pcm16' } on the pipeline or VoiceAgent for raw 24kHz PCM (see TTS options).

The package includes audio conversion utilities:

import {
  float32ToPcm16,
  pcm16ToFloat32,
  pcmToWav,
  wavToPcm,
  resample,
  calculateRMS,
  detectAudioFormat,
} from '@cogitator-ai/voice';

Voice ToolsCopy Anchor Link

voiceTools({ stt, tts }) returns two tools for text agents that handle audio as data: transcribe_audio ({ audioBase64, language? } → { text, language, duration }) and speak_text ({ text, voice?, format? } → { audioBase64, format }). They are plain { name, description, parameters, execute } objects with Zod parameters — wrap them with tool() from @cogitator-ai/core:

import { Agent, tool } from '@cogitator-ai/core';
import { voiceTools, DeepgramSTT, ElevenLabsTTS } from '@cogitator-ai/voice';

const [transcribe, speak] = voiceTools({
  stt: new DeepgramSTT({ apiKey: process.env.DEEPGRAM_API_KEY! }),
  tts: new ElevenLabsTTS({ apiKey: process.env.ELEVENLABS_API_KEY! }),
});

const agent = new Agent({
  name: 'media-assistant',
  model: 'openai/gpt-5.5',
  instructions: 'Transcribe audio the user sends and answer with speech when asked.',
  tools: [tool(transcribe), tool(speak)],
});

transcribeTool(stt) and speakTool(tts) create them one at a time.

Next StepsCopy Anchor Link

• STT Providers — configure speech recognition
• TTS Providers — configure speech synthesis
• VAD — detect when users start and stop speaking
• Pipeline Mode — batch and streaming voice processing
• Realtime Mode — native speech-to-speech APIs