Twilio Phone Calls

OverviewCopy Anchor Link

A phone call becomes a voice session of the agent through a bidirectional Twilio Media Stream. When a call comes in, Twilio asks your voice webhook what to do. The webhook answers with TwiML that connects the call to a WebSocket stream, and VoiceAgent serves that stream: the caller's audio goes to the agent, the agent's voice goes back to the call.

caller -> Twilio -> POST /voice (TwiML webhook) -> <Connect><Stream url="wss://.../twilio">
                 -> wss://.../twilio (media stream) <-> VoiceAgent <-> STT, agent, TTS

Both pipeline mode (STT -> agent -> TTS) and realtime mode (OpenAI Realtime, Gemini Live) work over a call.

Quick startCopy Anchor Link

import http from 'node:http';
import { Agent, Cogitator } from '@cogitator-ai/core';
import {
  VoiceAgent,
  DeepgramSTT,
  ElevenLabsTTS,
  EnergyVAD,
  TwilioSessionTokens,
  createCogitatorRunner,
  twilioStreamTwiml,
  validateTwilioWebhook,
} from '@cogitator-ai/voice';

const authToken = process.env.TWILIO_AUTH_TOKEN!;
const webhookUrl = 'https://voice.example.com/voice';
const streamUrl = 'wss://voice.example.com/twilio';

const cogitator = new Cogitator({ memory: { adapter: 'memory' } });
const agent = new Agent({
  name: 'phone-assistant',
  model: 'google/gemini-3.5-flash-lite',
  instructions: 'You answer phone calls. Speak in one or two short sentences.',
});

const voiceAgent = new VoiceAgent({
  mode: 'pipeline',
  agent: createCogitatorRunner(cogitator, agent),
  stt: new DeepgramSTT({ apiKey: process.env.DEEPGRAM_API_KEY! }),
  tts: new ElevenLabsTTS({ apiKey: process.env.ELEVENLABS_API_KEY! }),
  vad: () => new EnergyVAD({ threshold: 0.02, sampleRate: 8000 }),
  bargeIn: true,
});

const tokens = new TwilioSessionTokens();

const server = http.createServer((req, res) => {
  if (req.method !== 'POST' || req.url !== '/voice') return void res.writeHead(404).end();
  let body = '';
  req.setEncoding('utf8');
  req.on('data', (chunk: string) => (body += chunk));
  req.on('end', () => {
    const signature = req.headers['x-twilio-signature'];
    const valid = validateTwilioWebhook({
      authToken,
      signature: Array.isArray(signature) ? signature[0] : signature,
      url: webhookUrl,
      body,
    });
    if (!valid) return void res.writeHead(403).end();

    const params = new URLSearchParams(body);
    const token = tokens.issue({
      callSid: params.get('CallSid') ?? undefined,
      metadata: { from: params.get('From') ?? 'unknown' },
    });
    res.writeHead(200, { 'Content-Type': 'text/xml' });
    res.end(twilioStreamTwiml({ url: streamUrl, token }));
  });
});

voiceAgent.attachTwilio(server, { authToken, streamUrl, tokens });
server.listen(8080);

Point the voice webhook of your Twilio number at POST https://voice.example.com/voice. The full runnable version is examples/voice/05-twilio-phone-agent.ts.

SecurityCopy Anchor Link

A media stream is a WebSocket anyone can try to open, so each stream is checked twice before it reaches the agent.

The upgrade is signed. Twilio sends X-Twilio-Signature with the upgrade: an HMAC-SHA1 keyed with your Auth Token over the stream URL, with no parameters. attachTwilio checks it for streamUrl and for streamUrl with a trailing slash, compares in constant time, and refuses anything else with 403. The signature covers only the URL, so it is the same for every call and could be replayed by anyone who saw one: it keeps strangers out, and the session token below is what ties a stream to a call. A server that handles upgrades itself checks with verifyTwilioUpgrade(req, { authToken, streamUrl }) and then hands the socket to voiceAgent.handleTwilio(ws, { tokens }).

The stream carries a one-time token. tokens.issue() in the webhook creates a token, and twilioStreamTwiml({ token }) passes it as a <Parameter> of the <Stream> (Twilio allows no query string in the stream URL). It comes back in the start event's customParameters, and the stream is refused (closed with 1008) unless the token was issued, has not expired (ttl, one minute by default) and was not used before. Given the webhook's CallSid, a token works only for a stream of that call. What issue was given comes back in the call_start event as call.grant. At most maxPending tokens (10 000) wait to be used at once.

The stream must behave. A socket that sends no start within startTimeout (10 seconds) is closed with 1008, so sockets opened with a replayed signature cannot hold connections. Messages are capped at 64 KiB (maxPayload), a binary frame closes the stream with 1003, and malformed JSON, a malformed event or a second start close it with 1008. handleTwilio serves a socket your server accepted, so cap connections and message size there.

Tokens live in the memory of the process. When the webhook and the stream may hit different instances, pass authorize and check a store they share (sharedTokens here is yours, with a consume that answers like TwilioSessionTokens.consume):

voiceAgent.attachTwilio(server, {
  authToken,
  streamUrl,
  authorize: async (start) =>
    (await sharedTokens.consume(
      start.customParameters[TWILIO_SESSION_TOKEN_PARAMETER],
      start.callSid
    )) !== undefined,
});

The webhook is signed too. validateTwilioWebhook checks the signature over the full URL and the sorted form parameters, and for a JSON body (a URL with a bodySHA256 query parameter) the signature over the URL and the SHA-256 of the body. It accepts the URL with and without its default port. validateTwilioSignature and twilioSignature are the building blocks.

AudioCopy Anchor Link

Twilio streams G.711 μ-law at 8 kHz, mono, in both directions.

Pipeline modeCopy Anchor Link

A TTS provider of your own declares what it returns with rawSampleRates, such as { pcm16: 22050 }. A provider without it cannot serve a call, and attachTwilio says so.

Realtime modeCopy Anchor Link

The session runs with audioFormat: 'pcmu'. OpenAI Realtime takes and returns audio/pcmu natively. Gemini Live takes PCM16, so the adapter decodes and resamples the call's audio to 16 kHz, and turns the model's 24 kHz audio into 8 kHz μ-law.

The model's audio is marked every 200 ms and at the end of each turn. When the caller speaks (OpenAI's speech_started, or Gemini's transcription of the caller, which also comes after the model finished while its audio still plays), the call's audio is cleared, and the OpenAI answer is truncated with conversation.item.truncate at the last mark that was played (later audio items at 0, and their audio still on its way is dropped), so the model's context holds what the caller heard. Gemini drops an interrupted answer itself. Its transcripts of the caller come apart from the model's output, so while the model generates only Gemini's own interruption counts, and afterwards only a transcript that starts a new utterance (after one marked finished, or a pause of 1.5 seconds). A key press with dtmfInput interrupts the same way before its text is sent.

const voiceAgent = new VoiceAgent({
  mode: 'realtime',
  agent: createCogitatorRunner(cogitator, agent),
  realtimeProvider: 'openai',
  realtimeApiKey: process.env.OPENAI_API_KEY!,
});
voiceAgent.attachTwilio(server, { authToken, streamUrl, tokens });

What the caller heardCopy Anchor Link

Audio goes to Twilio faster than it is played, so "sent" is not "heard". After each spoken sentence the pipeline session sends a mark, and Twilio echoes it once the audio before it was played. A mark whose echo does not come within the duration of the audio before it plus markTimeoutMargin (3 seconds) counts as not played, so a lost echo cannot hang a call. The session uses that:

• A turn stays speaking until its last mark comes back, so in pipeline mode turn_end means the caller heard the whole answer, and a barge-in while Twilio is still playing stops it. In realtime mode turn_end means the model finished its answer.
• A barge-in (with bargeIn and a VAD) or an interruption sends clear, which drops the audio Twilio still holds. The marks still pending come back as not played.
• A streamed turn settles with the sentences whose marks were played, up to the first that was not. With createCogitatorRunner, the conversation keeps only those, marked metadata.interrupted.

A sentence cut in the middle by a barge-in counts as not heard, and so does a mark Twilio echoes after the clear was sent.

Key presses and the end of a callCopy Anchor Link

Key presses (DTMF) arrive as dtmf events. dtmfInput turns a key into text the agent gets as the caller's input:

voiceAgent.attachTwilio(server, {
  authToken,
  streamUrl,
  tokens,
  dtmfInput: (digit, call) => (digit === '0' ? 'Connect me to a person.' : undefined),
});
voiceAgent.on('dtmf', (sessionId, digit) => console.log(sessionId, digit));

When the caller hangs up, Twilio stops the stream, and the session ends with session_end. voiceAgent.close() ends every call, also those served with handleTwilio.

EventsCopy Anchor Link

TwiMLCopy Anchor Link

twilioStreamTwiml builds <Response><Connect><Stream> with escaped values:

twilioStreamTwiml({
  url: 'wss://voice.example.com/twilio',
  token: tokens.issue(),
  parameters: { language: 'en' },
  statusCallback: 'https://voice.example.com/stream-status',
  statusCallbackMethod: 'POST',
});

The URL must use wss:// and have no query string, and each parameter's name and value hold at most 500 characters together.

TwilioMediaStreamCopy Anchor Link

TwilioMediaStream is the stream on its own, for a custom setup. It reads Twilio's events (connected, start, media, mark, dtmf, stop, numbers sent as strings included), emits start, audio, dtmf, mark and stop, and sends audio back as base64 μ-law media messages of 20 ms (frameMs). inbound and outbound set the formats on your side, transcoded to and from μ-law at 8 kHz. It implements VoicePlayback: mark() resolves true once Twilio played the audio before it, and clear() drops what was not played.

const stream = new TwilioMediaStream(ws, {
  inbound: { encoding: 'pcm16', sampleRate: 16000 },
  outbound: { encoding: 'pcm16', sampleRate: 24000 },
  authorize: (start) =>
    tokens.consume(start.customParameters[TWILIO_SESSION_TOKEN_PARAMETER]) !== undefined,
});
stream.on('audio', (pcm16k) => recognizer.write(pcm16k));
stream.sendAudio(pcm24k);
const heard = await stream.mark();