# Twilio Phone Calls URL: https://cogitator.app/docs/voice/twilio > Answer phone calls with a Cogitator agent through Twilio Media Streams - signed upgrades, one-time session tokens, μ-law audio, barge-in and memory of only what the caller heard. ## Overview A phone call becomes a voice session of the agent through a bidirectional [Twilio Media Stream](https://www.twilio.com/docs/voice/media-streams). When a call comes in, Twilio asks your voice webhook what to do. The webhook answers with TwiML that connects the call to a WebSocket stream, and `VoiceAgent` serves that stream: the caller's audio goes to the agent, the agent's voice goes back to the call. ``` caller -> Twilio -> POST /voice (TwiML webhook) -> -> wss://.../twilio (media stream) <-> VoiceAgent <-> STT, agent, TTS ``` Both pipeline mode (STT -> agent -> TTS) and realtime mode (OpenAI Realtime, Gemini Live) work over a call. ## Quick start ```typescript import http from 'node:http'; import { Agent, Cogitator } from '@cogitator-ai/core'; import { VoiceAgent, DeepgramSTT, ElevenLabsTTS, EnergyVAD, TwilioSessionTokens, createCogitatorRunner, twilioStreamTwiml, validateTwilioWebhook, } from '@cogitator-ai/voice'; const authToken = process.env.TWILIO_AUTH_TOKEN!; const webhookUrl = 'https://voice.example.com/voice'; const streamUrl = 'wss://voice.example.com/twilio'; const cogitator = new Cogitator({ memory: { adapter: 'memory' } }); const agent = new Agent({ name: 'phone-assistant', model: 'google/gemini-3.5-flash-lite', instructions: 'You answer phone calls. Speak in one or two short sentences.', }); const voiceAgent = new VoiceAgent({ mode: 'pipeline', agent: createCogitatorRunner(cogitator, agent), stt: new DeepgramSTT({ apiKey: process.env.DEEPGRAM_API_KEY! }), tts: new ElevenLabsTTS({ apiKey: process.env.ELEVENLABS_API_KEY! }), vad: () => new EnergyVAD({ threshold: 0.02, sampleRate: 8000 }), bargeIn: true, }); const tokens = new TwilioSessionTokens(); const server = http.createServer((req, res) => { if (req.method !== 'POST' || req.url !== '/voice') return void res.writeHead(404).end(); let body = ''; req.setEncoding('utf8'); req.on('data', (chunk: string) => (body += chunk)); req.on('end', () => { const signature = req.headers['x-twilio-signature']; const valid = validateTwilioWebhook({ authToken, signature: Array.isArray(signature) ? signature[0] : signature, url: webhookUrl, body, }); if (!valid) return void res.writeHead(403).end(); const params = new URLSearchParams(body); const token = tokens.issue({ callSid: params.get('CallSid') ?? undefined, metadata: { from: params.get('From') ?? 'unknown' }, }); res.writeHead(200, { 'Content-Type': 'text/xml' }); res.end(twilioStreamTwiml({ url: streamUrl, token })); }); }); voiceAgent.attachTwilio(server, { authToken, streamUrl, tokens }); server.listen(8080); ``` Point the voice webhook of your Twilio number at `POST https://voice.example.com/voice`. The full runnable version is [`examples/voice/05-twilio-phone-agent.ts`](https://github.com/cogitator-ai/Cogitator-AI/tree/main/examples/voice/05-twilio-phone-agent.ts). ## Security A media stream is a WebSocket anyone can try to open, so each stream is checked twice before it reaches the agent. **The upgrade is signed.** Twilio sends `X-Twilio-Signature` with the upgrade: an HMAC-SHA1 keyed with your Auth Token over the stream URL, with no parameters. `attachTwilio` checks it for `streamUrl` and for `streamUrl` with a trailing slash, compares in constant time, and refuses anything else with 403. The signature covers only the URL, so it is the same for every call and could be replayed by anyone who saw one: it keeps strangers out, and the session token below is what ties a stream to a call. A server that handles upgrades itself checks with `verifyTwilioUpgrade(req, { authToken, streamUrl })` and then hands the socket to `voiceAgent.handleTwilio(ws, { tokens })`. **The stream carries a one-time token.** `tokens.issue()` in the webhook creates a token, and `twilioStreamTwiml({ token })` passes it as a `` of the `` (Twilio allows no query string in the stream URL). It comes back in the `start` event's `customParameters`, and the stream is refused (closed with 1008) unless the token was issued, has not expired (`ttl`, one minute by default) and was not used before. Given the webhook's `CallSid`, a token works only for a stream of that call. What `issue` was given comes back in the `call_start` event as `call.grant`. At most `maxPending` tokens (10 000) wait to be used at once. **The stream must behave.** A socket that sends no `start` within `startTimeout` (10 seconds) is closed with 1008, so sockets opened with a replayed signature cannot hold connections. Messages are capped at 64 KiB (`maxPayload`), a binary frame closes the stream with 1003, and malformed JSON, a malformed event or a second `start` close it with 1008. `handleTwilio` serves a socket your server accepted, so cap connections and message size there. Tokens live in the memory of the process. When the webhook and the stream may hit different instances, pass `authorize` and check a store they share (`sharedTokens` here is yours, with a `consume` that answers like `TwilioSessionTokens.consume`): ```typescript voiceAgent.attachTwilio(server, { authToken, streamUrl, authorize: async (start) => (await sharedTokens.consume( start.customParameters[TWILIO_SESSION_TOKEN_PARAMETER], start.callSid )) !== undefined, }); ``` **The webhook is signed too.** `validateTwilioWebhook` checks the signature over the full URL and the sorted form parameters, and for a JSON body (a URL with a `bodySHA256` query parameter) the signature over the URL and the SHA-256 of the body. It accepts the URL with and without its default port. `validateTwilioSignature` and `twilioSignature` are the building blocks. ## Audio Twilio streams G.711 μ-law at 8 kHz, mono, in both directions. ### Pipeline mode | Step | What happens | | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | STT | A provider that lists `mulaw` in `streamEncodings` gets the call's audio as is: Deepgram streams with `encoding=mulaw&sample_rate=8000`, OpenAI STT decodes it before upload. Any other gets 8 kHz PCM16. | | VAD | Required, since Twilio never says when the caller stopped speaking. Each call gets its own (a factory, or a `clone()` of the instance) and the audio decoded, resampled to its `sampleRate`. Give `SileroVAD` or `EnergyVAD` `sampleRate: 8000`. | | TTS | Asked for `mulaw` when `rawSampleRates` lists it at 8000 (ElevenLabs `ulaw_8000`), else for `pcm16` (OpenAI TTS at 24 kHz), which is resampled with a band-limited filter and encoded on the way out. | A TTS provider of your own declares what it returns with `rawSampleRates`, such as `{ pcm16: 22050 }`. A provider without it cannot serve a call, and `attachTwilio` says so. ### Realtime mode The session runs with `audioFormat: 'pcmu'`. OpenAI Realtime takes and returns `audio/pcmu` natively. Gemini Live takes PCM16, so the adapter decodes and resamples the call's audio to 16 kHz, and turns the model's 24 kHz audio into 8 kHz μ-law. The model's audio is marked every 200 ms and at the end of each turn. When the caller speaks (OpenAI's `speech_started`, or Gemini's transcription of the caller, which also comes after the model finished while its audio still plays), the call's audio is cleared, and the OpenAI answer is truncated with `conversation.item.truncate` at the last mark that was played (later audio items at 0, and their audio still on its way is dropped), so the model's context holds what the caller heard. Gemini drops an interrupted answer itself. Its transcripts of the caller come apart from the model's output, so while the model generates only Gemini's own interruption counts, and afterwards only a transcript that starts a new utterance (after one marked `finished`, or a pause of 1.5 seconds). A key press with `dtmfInput` interrupts the same way before its text is sent. ```typescript const voiceAgent = new VoiceAgent({ mode: 'realtime', agent: createCogitatorRunner(cogitator, agent), realtimeProvider: 'openai', realtimeApiKey: process.env.OPENAI_API_KEY!, }); voiceAgent.attachTwilio(server, { authToken, streamUrl, tokens }); ``` ## What the caller heard Audio goes to Twilio faster than it is played, so "sent" is not "heard". After each spoken sentence the pipeline session sends a `mark`, and Twilio echoes it once the audio before it was played. A mark whose echo does not come within the duration of the audio before it plus `markTimeoutMargin` (3 seconds) counts as not played, so a lost echo cannot hang a call. The session uses that: * A turn stays `speaking` until its last mark comes back, so in pipeline mode `turn_end` means the caller heard the whole answer, and a barge-in while Twilio is still playing stops it. In realtime mode `turn_end` means the model finished its answer. * A barge-in (with `bargeIn` and a VAD) or an interruption sends `clear`, which drops the audio Twilio still holds. The marks still pending come back as not played. * A streamed turn settles with the sentences whose marks were played, up to the first that was not. With `createCogitatorRunner`, the conversation keeps only those, marked `metadata.interrupted`. A sentence cut in the middle by a barge-in counts as not heard, and so does a mark Twilio echoes after the `clear` was sent. ## Key presses and the end of a call Key presses (DTMF) arrive as `dtmf` events. `dtmfInput` turns a key into text the agent gets as the caller's input: ```typescript voiceAgent.attachTwilio(server, { authToken, streamUrl, tokens, dtmfInput: (digit, call) => (digit === '0' ? 'Connect me to a person.' : undefined), }); voiceAgent.on('dtmf', (sessionId, digit) => console.log(sessionId, digit)); ``` When the caller hangs up, Twilio stops the stream, and the session ends with `session_end`. `voiceAgent.close()` ends every call, also those served with `handleTwilio`. ## Events | Event | Arguments | When | | --------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `call_start` | `call: TwilioCall` | A stream passed its checks: `sessionId`, `callSid`, `streamSid`, `accountSid`, `customParameters` (without the token), `grant` | | `session_start` | `sessionId` | The voice session of the call started | | `dtmf` | `sessionId`, `digit` | The caller pressed a key | | `session_end` | `sessionId` | The call ended | | `error` | `error` | A refused stream, a malformed event, or a provider error | ## TwiML `twilioStreamTwiml` builds `` with escaped values: ```typescript twilioStreamTwiml({ url: 'wss://voice.example.com/twilio', token: tokens.issue(), parameters: { language: 'en' }, statusCallback: 'https://voice.example.com/stream-status', statusCallbackMethod: 'POST', }); ``` The URL must use `wss://` and have no query string, and each parameter's name and value hold at most 500 characters together. ## TwilioMediaStream `TwilioMediaStream` is the stream on its own, for a custom setup. It reads Twilio's events (`connected`, `start`, `media`, `mark`, `dtmf`, `stop`, numbers sent as strings included), emits `start`, `audio`, `dtmf`, `mark` and `stop`, and sends audio back as base64 μ-law `media` messages of 20 ms (`frameMs`). `inbound` and `outbound` set the formats on your side, transcoded to and from μ-law at 8 kHz. It implements `VoicePlayback`: `mark()` resolves true once Twilio played the audio before it, and `clear()` drops what was not played. ```typescript const stream = new TwilioMediaStream(ws, { inbound: { encoding: 'pcm16', sampleRate: 16000 }, outbound: { encoding: 'pcm16', sampleRate: 24000 }, authorize: (start) => tokens.consume(start.customParameters[TWILIO_SESSION_TOKEN_PARAMETER]) !== undefined, }); stream.on('audio', (pcm16k) => recognizer.write(pcm16k)); stream.sendAudio(pcm24k); const heard = await stream.mark(); ```