# Test Utilities

URL: https://cogitator.app/docs/testing/test-utils

> @cogitator-ai/test-utils — a scriptable mock LLM backend, a recording memory adapter, fixtures and async/stream helpers for testing Cogitator code.

`@cogitator-ai/test-utils` gives you deterministic stand-ins for the parts of a run that are slow, paid or non-deterministic:

```bash
pnpm add -D @cogitator-ai/test-utils
```

| Export                                                  | What it is                                                                     |
| ------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `MockLLMBackend`, `createMockLLMBackend()`              | An `LLMBackend` that answers with scripted responses and records every request |
| `MockMemoryAdapter`, `createMockMemoryAdapter()`        | An in-process `MemoryAdapter` that records every method call                   |
| `createTestAgentConfig()`, `createTestAgent()`          | Agent configs and agent-shaped objects with test defaults                      |
| `createUserMessage()`, `createToolCall()`, …            | Message, tool call and tool result fixtures                                    |
| `createTestTool()`, `createCalculatorTool()`, …         | Tool-shaped fixtures with JSON Schema parameters                               |
| `createMockRunResult()`, …                              | `RunResult` fixtures for code that consumes run results                        |
| `collectStreamContent()`, …                             | Drain `ChatStreamChunk` streams                                                |
| `waitFor()`, `eventually()`, `delay()`, `withTimeout()` | Async test helpers                                                             |

## MockLLMBackend

A `Cogitator` runs agents on the backends you hand it in `llm.backends`: an agent with model `mock/<anything>` runs on the backend registered under `mock`, and the backend receives `<anything>` as `request.model`.

```typescript
import { Cogitator, Agent, tool } from '@cogitator-ai/core';
import { MockLLMBackend, createToolCall } from '@cogitator-ai/test-utils';
import { z } from 'zod';

const getWeather = tool({
  name: 'get_weather',
  description: 'Get the weather for a city',
  parameters: z.object({ city: z.string() }),
  execute: async ({ city }) => ({ city, temperature: 22 }),
});

const mock = new MockLLMBackend().setResponses([
  { toolCalls: [createToolCall('get_weather', { city: 'Tokyo' })] },
  { content: 'It is 22°C in Tokyo.' },
]);

const cog = new Cogitator({ llm: { backends: { mock } } });
const agent = new Agent({
  name: 'weather',
  model: 'mock/test-model',
  instructions: 'You are a weather assistant.',
  tools: [getWeather],
});

const result = await cog.run(agent, { input: 'Weather in Tokyo?' });

expect(result.output).toBe('It is 22°C in Tokyo.');
expect(result.toolCalls.map((call) => call.name)).toEqual(['get_weather']);
expect(mock.getCallCount()).toBe(2);
expect(mock.getLastCall()?.model).toBe('test-model');

await cog.close();
```

The tool really runs: the first response asks for `get_weather`, the runtime executes it and sends the result back, and the second response ends the run.

### Scripting Responses

| Method                                     | Effect                                                                         |
| ------------------------------------------ | ------------------------------------------------------------------------------ |
| `setResponse(response)`                    | Answer every `chat()` call with `response`                                     |
| `setResponses([r1, r2, …])`                | Answer calls in order; once the list is used up, the last response repeats     |
| `setToolCallResponse(toolCalls, content?)` | Shortcut for `setResponse({ content, toolCalls, finishReason: 'tool_calls' })` |
| `setDefaultDelay(ms)`                      | Wait `ms` before every response and stream chunk                               |
| `reset()`                                  | Forget responses, stream chunks and recorded calls                             |

All setters return the backend, so they chain. A `MockChatResponse` is a partial `ChatResponse` plus two test-only fields:

```typescript
interface MockChatResponse extends Partial<ChatResponse> {
  content?: string; // default ''
  toolCalls?: ToolCall[];
  finishReason?: 'stop' | 'tool_calls' | 'length' | 'error'; // default: 'tool_calls' with toolCalls, else 'stop'
  error?: Error; // thrown instead of answering
  delay?: number; // ms, overrides setDefaultDelay
}
```

Without any response set, `chat()` answers with empty content and `finishReason: 'stop'`. Usage is estimated (about four characters per token) unless the response sets `usage`.

An `error` response makes `chat()` throw. The runtime retries LLM calls by default, so for failure tests turn retries off (or expect `maxRetries + 1` calls):

```typescript
const mock = new MockLLMBackend().setResponse({ error: new Error('provider down') });
const cog = new Cogitator({ llm: { backends: { mock }, retry: false } });

await expect(cog.run(agent, { input: 'hi' })).rejects.toThrow('provider down');
```

### Streaming

`chatStream()` plays scripted chunks; it never uses the `chat()` responses. The runtime streams only when the run has both `stream: true` and `onToken`:

```typescript
const mock = new MockLLMBackend().setStreamChunks([
  { content: 'Once ' },
  { content: 'upon a time' },
]);
const cog = new Cogitator({ llm: { backends: { mock } } });

const tokens: string[] = [];
const result = await cog.run(agent, {
  input: 'Tell me a story',
  stream: true,
  onToken: (token) => tokens.push(token),
});

expect(tokens).toEqual(['Once ', 'upon a time']);
expect(result.output).toBe('Once upon a time');
```

`setMultipleStreamChunks([[…], […]])` scripts one chunk list per `chatStream()` call (the last list repeats), so a streamed tool call followed by a streamed answer looks like this:

```typescript
mock.setMultipleStreamChunks([
  [{ toolCalls: [{ id: 'call_1', name: 'get_weather', arguments: { city: 'Oslo' } }] }],
  [{ content: 'It is cold in Oslo.' }],
]);
```

The last chunk of each list carries `finishReason` (default `'stop'`) and usage.

### Inspecting Requests

| Method                     | Returns                                                    |
| -------------------------- | ---------------------------------------------------------- |
| `getCalls()`               | Every `ChatRequest` received (`chat()` and `chatStream()`) |
| `getLastCall()`            | The most recent request, or `undefined`                    |
| `getCallCount()`           | Number of requests                                         |
| `wasCalledWith(predicate)` | Whether any request matches `(request) => boolean`         |

Each request is recorded as a snapshot of what was sent at that moment: the runtime keeps appending to the same conversation, but `getCalls()[0].messages` still holds only the messages of the first call, and later calls show the tool results added in between.

```typescript
expect(mock.getCalls()[0].tools?.map((t) => t.name)).toEqual(['get_weather']);
expect(
  mock.wasCalledWith((req) => req.messages[0].content === 'You are a weather assistant.')
).toBe(true);
```

`provider` of the mock is always `'openai'`; routing goes by the `llm.backends` key, not by this field.

### Your Own Scripted Backend

When responses depend on the request (who is asking, which tool results came back), implement `LLMBackend` directly — that is what the core package's own tests do:

```typescript
import type { ChatRequest, ChatResponse, LLMBackend } from '@cogitator-ai/types';

const requests: ChatRequest[] = [];
const backend: LLMBackend = {
  provider: 'openai',
  async chat(request): Promise<ChatResponse> {
    requests.push({ ...request, messages: [...request.messages] });
    const usage = { inputTokens: 1, outputTokens: 1, totalTokens: 2 };
    const hasToolResult = request.messages.some((m) => m.role === 'tool');
    return hasToolResult
      ? { id: 'r2', content: 'done', finishReason: 'stop', usage }
      : {
          id: 'r1',
          content: '',
          toolCalls: [{ id: 'call_1', name: 'get_weather', arguments: { city: 'Paris' } }],
          finishReason: 'tool_calls',
          usage,
        };
  },
  async *chatStream() {
    yield { id: 's', delta: {}, finishReason: 'stop' };
  },
};

const cog = new Cogitator({ llm: { backends: { mock: backend } } });
```

Copy `messages` on the way in, as `MockLLMBackend` does, so each recorded request keeps what was sent at that moment rather than the conversation as it ended.

## MockMemoryAdapter

An in-process `MemoryAdapter` (`provider: 'memory'`) that keeps threads and entries in maps and records every call. Hand it to a `Cogitator` through the `memory` setter:

```typescript
import { MockMemoryAdapter, MockLLMBackend } from '@cogitator-ai/test-utils';

const memory = new MockMemoryAdapter();
await memory.connect();

const mock = new MockLLMBackend().setResponse({ content: 'Hi there' });
const cog = new Cogitator({ llm: { backends: { mock } } });
cog.memory = memory;

await cog.run(agent, { input: 'Hello', threadId: 'thread-1' });
await cog.run(agent, { input: 'Again', threadId: 'thread-1' });

expect(memory.getCallsFor('createThread')).toHaveLength(1);
expect(memory.getCallsFor('addEntry').length).toBeGreaterThan(0);
```

Besides the `MemoryAdapter` methods it has `getCalls()` (`{ method, args }[]`), `getCallsFor(method)`, `isConnected()` and `reset()` (clears threads, entries, calls and the connection flag). `cog.close()` disconnects it.

## Fixtures

### Agents

```typescript
import { Agent } from '@cogitator-ai/core';
import { createTestAgentConfig, createTestAgent } from '@cogitator-ai/test-utils';

const agent = new Agent(createTestAgentConfig({ tools: [getWeather] }));
```

`createTestAgentConfig(overrides?)` returns an `AgentConfig` with a random `id`, `name: 'test-agent'`, `model: 'mock/test-model'`, `instructions: 'You are a helpful test assistant.'`, `temperature: 0.7`, `maxIterations: 10`, `timeout: 30000` and no tools — the model fits a backend registered as `mock`. Wrap it in `new Agent()` to run it.

`createTestAgent(overrides?)` returns a plain object implementing the `Agent` interface from `@cogitator-ai/types` (with `clone()` and `serialize()`), for code that only needs something agent-shaped. It is not an instance of the core `Agent` class.

### Messages, Tool Calls, Run Results

| Fixture                                                                                                                 | Returns                                                   |
| ----------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `createUserMessage(text)` / `createAssistantMessage(text)` / `createSystemMessage(text)`                                | `Message` with that role                                  |
| `createToolMessage(toolCallId, content, name?)`                                                                         | `Message` with `role: 'tool'`                             |
| `createMultimodalMessage(parts)` / `createImageUrlMessage(text, url)`                                                   | User message with content parts                           |
| `createConversation(...{ role, content })`                                                                              | `Message[]` of user and assistant turns                   |
| `createToolCall(name, args?)`                                                                                           | `ToolCall` with a random `call_…` id                      |
| `createToolResult(toolCall, result, error?)`                                                                            | `ToolResult` answering that call                          |
| `createMockRunResult(output, options?)`                                                                                 | A complete `RunResult` (usage 100/50 tokens, empty trace) |
| `createRunResultWithToolCalls(output, toolCalls, options?)` / `createRunResultWithMessages(output, messages, options?)` | `RunResult` with those fields                             |

```typescript
const call = createToolCall('get_weather', { city: 'Tokyo' });
const history = [
  createUserMessage('Weather in Tokyo?'),
  createToolMessage(call.id, JSON.stringify({ temperature: 22 }), 'get_weather'),
];
const result = createMockRunResult('It is 22°C.', { toolCalls: [call], messages: history });
```

### Tools

`createTestTool(options?)`, `createCalculatorTool()` (name `calculator`, `a op b` only), `createWeatherTool()` (name `get_weather`, always 22°C and sunny), `createFailingTool(message?)` (throws), `createSlowTool(delayMs?)` and `createToolSchema(name, description?)` build tool-shaped objects (`SimpleTool`) whose `parameters` are JSON Schema rather than Zod. Use them where code takes tool definitions or schemas, or call `execute()` directly:

```typescript
const calculator = createCalculatorTool();
await expect(calculator.execute({ expression: '6 * 7' }, context)).resolves.toEqual({ result: 42 });
```

They are not `Tool` objects from `tool()` (no Zod schema, no `toJSON()`). For agent runs, define tools with `tool()` as usual.

## Helpers

| Helper                                           | Behaviour                                                                   |
| ------------------------------------------------ | --------------------------------------------------------------------------- |
| `collectStreamContent(stream)`                   | Concatenates `delta.content` of a `ChatStreamChunk` generator               |
| `collectStreamChunks(stream)`                    | All chunks as an array                                                      |
| `collectStreamToolCalls(stream)`                 | `{ name, arguments }` of every complete streamed tool call                  |
| `createMockReadableStream(chunks)`               | A `ReadableStream<Uint8Array>` of the encoded strings, e.g. for SSE parsers |
| `waitFor(condition, { timeout?, interval? })`    | Polls until `condition()` is truthy and returns it (defaults 5000 / 50 ms)  |
| `eventually(assertion, { timeout?, interval? })` | Retries an assertion until it passes, then rethrows the last failure        |
| `delay(ms)`                                      | Resolves after `ms`                                                         |
| `withTimeout(promise, ms)`                       | Rejects with `Operation timed out after …ms` if `promise` is slower         |

```typescript
const mock = new MockLLMBackend().setStreamChunks([{ content: 'Hel' }, { content: 'lo' }]);
const text = await collectStreamContent(mock.chatStream({ model: 'm', messages: [] }));
expect(text).toBe('Hello');

const job = startBackgroundJob();
await eventually(() => expect(job.done).toBe(true), { timeout: 2000 });
```