# Vision & Audio

URL: https://cogitator.app/docs/core/vision-and-audio

> Multi-modal capabilities including image analysis, image generation, audio transcription, and text-to-speech.

## Overview

Cogitator provides four multi-modal tool factories for working with images and audio. Each factory returns a standard tool that can be given to any agent.

```typescript
import {
  createAnalyzeImageTool,
  createGenerateImageTool,
  createTranscribeAudioTool,
  createGenerateSpeechTool,
} from '@cogitator-ai/core';
```

## Image Analysis

`createAnalyzeImageTool` creates a tool that sends images to a vision-capable model for analysis. It supports both URL and base64 image inputs.

```typescript
import { Cogitator, Agent, createAnalyzeImageTool, OpenAIBackend } from '@cogitator-ai/core';

const openai = new OpenAIBackend({ apiKey: process.env.OPENAI_API_KEY! });

const analyzeImage = createAnalyzeImageTool({
  llm: openai,
  defaultModel: 'gpt-6.1-sol',
});

const agent = new Agent({
  name: 'vision-assistant',
  model: 'openai/gpt-6.1-sol',
  instructions: 'You can analyze images. Use the analyzeImage tool when given an image URL.',
  tools: [analyzeImage],
});
```

### Parameters

| Parameter | Type                           | Description                                     |
| --------- | ------------------------------ | ----------------------------------------------- |
| `image`   | `string \| { data, mimeType }` | Image URL, or base64 data with its MIME type    |
| `prompt`  | `string`                       | Question or instruction about the image         |
| `detail`  | `'auto' \| 'low' \| 'high'`    | Analysis detail level (`high` uses more tokens) |
| `model`   | `string`                       | Override the vision model                       |

### Inline Images via RunOptions

You can also pass images directly through `RunOptions` without needing the analyze tool:

```typescript
const result = await cog.run(agent, {
  input: 'What do you see in this image?',
  images: [
    'https://example.com/photo.jpg',
    {
      data: base64EncodedImage,
      mimeType: 'image/png',
    },
  ],
});
```

The `images` array accepts URLs as strings or objects with `data` (base64) and `mimeType` fields. Supported MIME types: `image/jpeg`, `image/png`, `image/gif`, `image/webp`.

## Image Generation

`createGenerateImageTool` creates images with OpenAI image models. The default model is `gpt-image-2.5-flare`.

```typescript
const generateImage = createGenerateImageTool({
  apiKey: process.env.OPENAI_API_KEY!,
});

const agent = new Agent({
  name: 'artist',
  model: 'openai/gpt-6.1-sol',
  instructions: 'You create images based on user descriptions. Use the generateImage tool.',
  tools: [generateImage],
});
```

### Configuration

| Option    | Type     | Default                     | Description                                                      |
| --------- | -------- | --------------------------- | ---------------------------------------------------------------- |
| `apiKey`  | `string` | `OPENAI_API_KEY`            | OpenAI API key                                                   |
| `baseUrl` | `string` | `https://api.openai.com/v1` | API base URL (for OpenAI-compatible image endpoints)             |
| `model`   | `string` | `gpt-image-2.5-flare`       | Image model; `dall-e-*` ids keep the legacy DALL-E request shape |

Set `model` to a `dall-e-*` id only for OpenAI-compatible servers that still serve DALL-E. In that mode the tool sends `size` (`auto` becomes `1024x1024`), `quality` (`hd` or `standard`), `style` and `response_format: 'url'`, and the result carries a hosted `url` instead of base64 data.

### Parameters

| Parameter      | Type                                                                                | Description                                                                             |
| -------------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `prompt`       | `string`                                                                            | Detailed description of the image to generate                                           |
| `size`         | `'auto' \| '1024x1024' \| '1536x1024' \| '1024x1536' \| '1792x1024' \| '1024x1792'` | Image dimensions (default: `auto`, the model chooses)                                   |
| `quality`      | `'auto' \| 'low' \| 'medium' \| 'high' \| 'xhigh' \| 'max' \| 'standard' \| 'hd'`   | Rendering quality (default: `auto`); legacy `standard` maps to `medium`, `hd` to `high` |
| `style`        | `'vivid' \| 'natural'`                                                              | Legacy DALL-E style; ignored by `gpt-image-*` models (describe the style in the prompt) |
| `outputFormat` | `'png' \| 'jpeg' \| 'webp'`                                                         | Image file format (default: `png`)                                                      |
| `background`   | `'auto' \| 'transparent' \| 'opaque'`                                               | Background handling; `transparent` requires `png` or `webp`                             |

### Result

```typescript
{
  imageBase64: 'iVBORw0KGgo...', // base64-encoded image bytes (gpt-image-* models)
  mimeType: 'image/png',
  model: 'gpt-image-2.5-flare',
  size: 'auto',
  quality: 'auto',
}
```

`url` is optional and present only when a legacy `dall-e-*` compatible endpoint returns a hosted image; `style` is included only in that mode. `revisedPrompt` is set when the API returns one.

## Audio Transcription

`createTranscribeAudioTool` transcribes audio to text using OpenAI transcription models (`gpt-transcribe` by default).

```typescript
const transcribe = createTranscribeAudioTool({
  apiKey: process.env.OPENAI_API_KEY!,
  defaultModel: 'gpt-transcribe',
  defaultLanguage: 'en',
});

const agent = new Agent({
  name: 'transcriber',
  model: 'openai/gpt-6.1-sol',
  instructions: 'Transcribe audio files. Use the transcribeAudio tool.',
  tools: [transcribe],
});
```

### Parameters

| Parameter    | Type                                                                                 | Description                                           |
| ------------ | ------------------------------------------------------------------------------------ | ----------------------------------------------------- |
| `audio`      | `string \| { data, format }`                                                         | URL or base64-encoded audio data                      |
| `language`   | `string`                                                                             | ISO-639-1 code (`'en'`, `'es'`, `'fr'`, `'ja'`, etc.) |
| `model`      | `'gpt-transcribe' \| 'whisper-1' \| 'gpt-4o-transcribe' \| 'gpt-4o-mini-transcribe'` | Transcription model                                   |
| `timestamps` | `boolean`                                                                            | Include word-level timestamps (see below)             |

Supported audio formats: `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `wav`, `webm`, `ogg`, `flac` (up to 25MB).

The default model is `gpt-transcribe`. Only `whisper-1` returns word timestamps, so when `timestamps: true` is requested without an explicit `model` (and no `defaultModel` is configured), the tool uses `whisper-1`. `words` and `duration` are returned only for `whisper-1` requests with timestamps.

### Result

```typescript
{
  text: 'The transcribed content of the audio file...',
  language: 'en',
  duration: 42.5,
  words: [
    { word: 'The', start: 0.0, end: 0.1 },
    { word: 'transcribed', start: 0.1, end: 0.5 },
    // ...
  ],
}
```

### Inline Audio via RunOptions

Audio files can be passed directly to the run. Each file is transcribed with OpenAI `gpt-transcribe` before the agent sees the input, and the transcripts are prepended to `input` as `[Audio transcription]: ...` (or `[Audio N transcription]: ...` for several files). Transcription uses `llm.providers.openai.apiKey`, falling back to `OPENAI_API_KEY`; the run fails with a clear error when neither is set. Prompt-injection detection and input guardrails run on the combined text.

```typescript
const result = await cog.run(agent, {
  input: 'Summarize what was said in this recording.',
  audio: ['https://example.com/meeting.mp3', { data: base64Audio, format: 'wav' }],
});
```

## Text-to-Speech

`createGenerateSpeechTool` converts text to natural-sounding speech using OpenAI TTS. The default model is `gpt-4o-mini-tts`; `tts-1` and `tts-1-hd` are legacy models.

```typescript
const speak = createGenerateSpeechTool({
  apiKey: process.env.OPENAI_API_KEY!,
  defaultModel: 'gpt-4o-mini-tts',
  defaultVoice: 'nova',
  defaultFormat: 'mp3',
});

const agent = new Agent({
  name: 'narrator',
  model: 'openai/gpt-6.1-sol',
  instructions: 'Read text aloud using the generateSpeech tool.',
  tools: [speak],
});
```

### Parameters

| Parameter | Type                                                   | Description                                |
| --------- | ------------------------------------------------------ | ------------------------------------------ |
| `text`    | `string`                                               | Text to convert (max 4096 characters)      |
| `voice`   | `string`                                               | Voice selection (see below)                |
| `model`   | `'tts-1' \| 'tts-1-hd' \| 'gpt-4o-mini-tts'`           | TTS model                                  |
| `speed`   | `number`                                               | Playback speed, 0.25 to 4.0 (default: 1.0) |
| `format`  | `'mp3' \| 'opus' \| 'aac' \| 'flac' \| 'wav' \| 'pcm'` | Output format                              |

### Available Voices

`alloy`, `ash`, `ballad`, `coral`, `echo`, `fable`, `nova`, `onyx`, `sage`, `shimmer`, `verse`, `marin`, `cedar`

### Result

```typescript
{
  audioBase64: 'AAAA...',  // base64-encoded audio
  format: 'mp3',
  voice: 'nova',
  model: 'gpt-4o-mini-tts',
  textLength: 256,
}
```

## Utilities

Cogitator exports helpers for working with image and audio data:

```typescript
import {
  fetchImageAsBase64,
  fetchAudioAsBuffer,
  audioInputToBuffer,
  isValidAudioFormat,
  getAudioMimeType,
} from '@cogitator-ai/core';

const image = await fetchImageAsBase64('https://example.com/photo.jpg');
// { data: 'base64...', mediaType: 'image/jpeg' } — up to 20 MB, 30 s timeout

const audio = await fetchAudioAsBuffer('https://example.com/audio.mp3');
// { buffer: Buffer, format: 'mp3', filename: 'audio.mp3' } — 60 s timeout

const input = await audioInputToBuffer({ data: base64Audio, format: 'wav' }); // URL or base64 AudioInput
isValidAudioFormat('ogg'); // true
getAudioMimeType('mp3'); // 'audio/mpeg'
```

Both fetchers take `{ timeout, signal }` as a second argument.