Streaming

OverviewCopy Anchor Link

Cogitator uses a send-then-edit pattern for streaming responses in chat. Instead of waiting for the full LLM response (2-10 seconds), the user sees the answer appearing progressively.

How It WorksCopy Anchor Link

Streaming is on when the Gateway gets a stream config; without it every answer is sent once, complete (chunked to the platform limit).

1. LLM generates tokens via cogitator.run() with stream: true
2. Tokens accumulate in a StreamBuffer
3. Every flushInterval ms (measured from the end of the previous flush), the buffer flushes:
   • First flush → channel.sendText() creates a new message (or a Telegram draft via sendDraft)
   • Subsequent flushes → channel.editText() updates the message
4. When the text reaches maxMessageChars (platform limit by default) the current message is finalized at a line/word boundary and a new one is started — no text is lost
5. After generation completes → final edit (or final message in draft mode)

All channel calls are serialized, so a slow sendText never races with an edit and every created message is tracked (for deleteOnAbort).

ConfigurationCopy Anchor Link

interface StreamConfig {
  flushInterval: number; // ms between flushes, e.g. 500
  minChunkSize: number; // skip a flush while the current message text is shorter than this
  minInitialChars?: number; // hold back the first message until N chars
  maxMessageChars?: number; // split threshold, default: platform limit
  deleteOnAbort?: boolean; // delete partial messages when the run fails or is interrupted
}

const gateway = new Gateway({
  // ...
  stream: {
    flushInterval: 500,
    minChunkSize: 20,
  },
});

Platform SupportCopy Anchor Link

RuntimeBuilder streams with flushInterval: 600 and minChunkSize: 30 unless stream in cogitator.yml overrides them. Telegram streams through rich drafts (sendRichMessageDraft, Markdown rendered as written) with a stop button, and falls back to editMessageText in groups or if a draft fails; a press of the stop button stops the run and keeps what was written as the reply (stream.stopButton: false turns it off); Discord edits keep their continuation messages in sync when an answer grows past 2,000 chars.

If the model produced no tokens (e.g. a provider without streaming), the Gateway sends the complete response as regular chunked messages.

StreamBuffer APICopy Anchor Link

The StreamBuffer class is used internally by Gateway, but can be used standalone:

import { StreamBuffer, adaptMarkdown } from '@cogitator-ai/channels';

const buffer = new StreamBuffer(
  channel, // any Channel implementation
  chatId,
  { flushInterval: 500, minChunkSize: 20, maxMessageChars: 4096 },
  replyToMessageId, // optional, used for the first message only
  true, // use channel.sendDraft when available
  (text) => adaptMarkdown(text, 'telegram') // optional formatter applied to every send/edit
);

buffer.start();

// append tokens as they arrive
buffer.append('Hello');
buffer.append(' world');
buffer.append('!');

// when generation is done — rethrows if the final delivery failed
const lastMessageId = await buffer.finish();
buffer.getMessageIds(); // every message created for this answer

// or, on failure
await buffer.abort(); // deletes partial messages when deleteOnAbort is set

forceNewMessage() finalizes the current message with the buffered text and continues in a new one.

Markdown FormattingCopy Anchor Link

The Gateway passes adaptMarkdown(text, channelType) as the formatter, so every flush is converted for the platform (code spans and fences are never touched):

• Telegram: # Heading and **bold** → *bold*, * item → • item
• Discord: Standard markdown passes through
• Slack: **bold** → *bold*, ~~strike~~ → ~strike~, links → <url|text>, &<> escaped
• WhatsApp: **bold** → *bold*, headings → bold, links → text (url)

Non-streamed answers are chunked with chunkMessage() at paragraph → line → word boundaries, closing and reopening code fences across chunks.