Vision & Audio
Multi-modal capabilities including image analysis, image generation, audio transcription, and text-to-speech.
Overview
Cogitator provides four multi-modal tool factories for working with images and audio. Each factory returns a standard tool that can be given to any agent.
import {
createAnalyzeImageTool,
createGenerateImageTool,
createTranscribeAudioTool,
createGenerateSpeechTool,
} from '@cogitator-ai/core';Image Analysis
createAnalyzeImageTool creates a tool that sends images to a vision-capable model for analysis. It supports both URL and base64 image inputs.
import { Cogitator, Agent, createAnalyzeImageTool, OpenAIBackend } from '@cogitator-ai/core';
const openai = new OpenAIBackend({ apiKey: process.env.OPENAI_API_KEY! });
const analyzeImage = createAnalyzeImageTool({
llm: openai,
defaultModel: 'gpt-6.1-sol',
});
const agent = new Agent({
name: 'vision-assistant',
model: 'openai/gpt-6.1-sol',
instructions: 'You can analyze images. Use the analyzeImage tool when given an image URL.',
tools: [analyzeImage],
});Parameters
| Parameter | Type | Description |
|---|---|---|
image | string | { data, mimeType } | Image URL, or base64 data with its MIME type |
prompt | string | Question or instruction about the image |
detail | 'auto' | 'low' | 'high' | Analysis detail level (high uses more tokens) |
model | string | Override the vision model |
Inline Images via RunOptions
You can also pass images directly through RunOptions without needing the analyze tool:
const result = await cog.run(agent, {
input: 'What do you see in this image?',
images: [
'https://example.com/photo.jpg',
{
data: base64EncodedImage,
mimeType: 'image/png',
},
],
});The images array accepts URLs as strings or objects with data (base64) and mimeType fields. Supported MIME types: image/jpeg, image/png, image/gif, image/webp.
Image Generation
createGenerateImageTool creates images with OpenAI image models. The default model is gpt-image-2.5-flare.
const generateImage = createGenerateImageTool({
apiKey: process.env.OPENAI_API_KEY!,
});
const agent = new Agent({
name: 'artist',
model: 'openai/gpt-6.1-sol',
instructions: 'You create images based on user descriptions. Use the generateImage tool.',
tools: [generateImage],
});Configuration
| Option | Type | Default | Description |
|---|---|---|---|
apiKey | string | OPENAI_API_KEY | OpenAI API key |
baseUrl | string | https://api.openai.com/v1 | API base URL (for OpenAI-compatible image endpoints) |
model | string | gpt-image-2.5-flare | Image model; dall-e-* ids keep the legacy DALL-E request shape |
Set model to a dall-e-* id only for OpenAI-compatible servers that still serve DALL-E. In that mode the tool sends size (auto becomes 1024x1024), quality (hd or standard), style and response_format: 'url', and the result carries a hosted url instead of base64 data.
Parameters
| Parameter | Type | Description |
|---|---|---|
prompt | string | Detailed description of the image to generate |
size | 'auto' | '1024x1024' | '1536x1024' | '1024x1536' | '1792x1024' | '1024x1792' | Image dimensions (default: auto, the model chooses) |
quality | 'auto' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' | 'standard' | 'hd' | Rendering quality (default: auto); legacy standard maps to medium, hd to high |
style | 'vivid' | 'natural' | Legacy DALL-E style; ignored by gpt-image-* models (describe the style in the prompt) |
outputFormat | 'png' | 'jpeg' | 'webp' | Image file format (default: png) |
background | 'auto' | 'transparent' | 'opaque' | Background handling; transparent requires png or webp |
Result
{
imageBase64: 'iVBORw0KGgo...', // base64-encoded image bytes (gpt-image-* models)
mimeType: 'image/png',
model: 'gpt-image-2.5-flare',
size: 'auto',
quality: 'auto',
}url is optional and present only when a legacy dall-e-* compatible endpoint returns a hosted image; style is included only in that mode. revisedPrompt is set when the API returns one.
Audio Transcription
createTranscribeAudioTool transcribes audio to text using OpenAI transcription models (gpt-transcribe by default).
const transcribe = createTranscribeAudioTool({
apiKey: process.env.OPENAI_API_KEY!,
defaultModel: 'gpt-transcribe',
defaultLanguage: 'en',
});
const agent = new Agent({
name: 'transcriber',
model: 'openai/gpt-6.1-sol',
instructions: 'Transcribe audio files. Use the transcribeAudio tool.',
tools: [transcribe],
});Parameters
| Parameter | Type | Description |
|---|---|---|
audio | string | { data, format } | URL or base64-encoded audio data |
language | string | ISO-639-1 code ('en', 'es', 'fr', 'ja', etc.) |
model | 'gpt-transcribe' | 'whisper-1' | 'gpt-4o-transcribe' | 'gpt-4o-mini-transcribe' | Transcription model |
timestamps | boolean | Include word-level timestamps (see below) |
Supported audio formats: mp3, mp4, mpeg, mpga, m4a, wav, webm, ogg, flac (up to 25MB).
The default model is gpt-transcribe. Only whisper-1 returns word timestamps, so when timestamps: true is requested without an explicit model (and no defaultModel is configured), the tool uses whisper-1. words and duration are returned only for whisper-1 requests with timestamps.
Result
{
text: 'The transcribed content of the audio file...',
language: 'en',
duration: 42.5,
words: [
{ word: 'The', start: 0.0, end: 0.1 },
{ word: 'transcribed', start: 0.1, end: 0.5 },
// ...
],
}Inline Audio via RunOptions
Audio files can be passed directly to the run. Each file is transcribed with OpenAI gpt-transcribe before the agent sees the input, and the transcripts are prepended to input as [Audio transcription]: ... (or [Audio N transcription]: ... for several files). Transcription uses llm.providers.openai.apiKey, falling back to OPENAI_API_KEY; the run fails with a clear error when neither is set. Prompt-injection detection and input guardrails run on the combined text.
const result = await cog.run(agent, {
input: 'Summarize what was said in this recording.',
audio: ['https://example.com/meeting.mp3', { data: base64Audio, format: 'wav' }],
});Text-to-Speech
createGenerateSpeechTool converts text to natural-sounding speech using OpenAI TTS. The default model is gpt-4o-mini-tts; tts-1 and tts-1-hd are legacy models.
const speak = createGenerateSpeechTool({
apiKey: process.env.OPENAI_API_KEY!,
defaultModel: 'gpt-4o-mini-tts',
defaultVoice: 'nova',
defaultFormat: 'mp3',
});
const agent = new Agent({
name: 'narrator',
model: 'openai/gpt-6.1-sol',
instructions: 'Read text aloud using the generateSpeech tool.',
tools: [speak],
});Parameters
| Parameter | Type | Description |
|---|---|---|
text | string | Text to convert (max 4096 characters) |
voice | string | Voice selection (see below) |
model | 'tts-1' | 'tts-1-hd' | 'gpt-4o-mini-tts' | TTS model |
speed | number | Playback speed, 0.25 to 4.0 (default: 1.0) |
format | 'mp3' | 'opus' | 'aac' | 'flac' | 'wav' | 'pcm' | Output format |
Available Voices
alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse, marin, cedar
Result
{
audioBase64: 'AAAA...', // base64-encoded audio
format: 'mp3',
voice: 'nova',
model: 'gpt-4o-mini-tts',
textLength: 256,
}Utilities
Cogitator exports helpers for working with image and audio data:
import {
fetchImageAsBase64,
fetchAudioAsBuffer,
audioInputToBuffer,
isValidAudioFormat,
getAudioMimeType,
} from '@cogitator-ai/core';
const image = await fetchImageAsBase64('https://example.com/photo.jpg');
// { data: 'base64...', mediaType: 'image/jpeg' } — up to 20 MB, 30 s timeout
const audio = await fetchAudioAsBuffer('https://example.com/audio.mp3');
// { buffer: Buffer, format: 'mp3', filename: 'audio.mp3' } — 60 s timeout
const input = await audioInputToBuffer({ data: base64Audio, format: 'wav' }); // URL or base64 AudioInput
isValidAudioFormat('ogg'); // true
getAudioMimeType('mp3'); // 'audio/mpeg'Both fetchers take { timeout, signal } as a second argument.