Cogitator
Testing

Testing Overview

Testing strategies for Cogitator tools, agents, workflows and swarms without calling a real LLM.

What to Test

Model output is non-deterministic, but almost everything around it is not. Test the deterministic parts directly and script the model:

  • Tools — call execute() with parameters and a ToolContext
  • Agent behaviour — script the LLM with a mock backend and assert on tool calls, output and the requests sent
  • Workflow logic — run DAGs of function nodes; no LLM needed
  • Error handling — make the mock throw and check how the run fails or recovers
  • Real providers — a smaller end-to-end suite against Google, OpenAI or Ollama

Unit Testing Tools

tool() returns a plain object, so its execute is directly callable. It takes the validated parameters and a ToolContext:

import { tool } from '@cogitator-ai/core';
import type { ToolContext } from '@cogitator-ai/types';
import { z } from 'zod';
import { describe, it, expect } from 'vitest';

const add = tool({
  name: 'add',
  description: 'Add two numbers',
  parameters: z.object({ a: z.number(), b: z.number() }),
  execute: async ({ a, b }) => ({ sum: a + b }),
});

const context: ToolContext = {
  agentId: 'agent-test',
  runId: 'run-test',
  signal: new AbortController().signal,
};

describe('add', () => {
  it('adds', async () => {
    await expect(add.execute({ a: 2, b: 2 }, context)).resolves.toEqual({ sum: 4 });
  });

  it('rejects bad arguments the way the runtime does', () => {
    expect(add.parameters.safeParse({ a: '2' }).success).toBe(false);
  });
});

execute does not validate on its own — the runtime parses the model's arguments with parameters first and answers the model with Invalid arguments: … when they do not match. The built-in tools work the same way (calculator.execute({ expression: '2 + 2' }, context)).

Agent Tests with a Mock Backend

Give the Cogitator a scripted backend in llm.backends and point the agent's model at it. @cogitator-ai/test-utils ships one:

import { Cogitator, Agent } from '@cogitator-ai/core';
import { MockLLMBackend, createToolCall } from '@cogitator-ai/test-utils';

const mock = new MockLLMBackend().setResponses([
  { toolCalls: [createToolCall('add', { a: 2, b: 3 })] },
  { content: 'The answer is 5.' },
]);

const cog = new Cogitator({ llm: { backends: { mock } } });
const agent = new Agent({
  name: 'math',
  model: 'mock/test-model',
  instructions: 'Use the add tool for arithmetic.',
  tools: [add],
});

const result = await cog.run(agent, { input: 'What is 2 + 3?' });

expect(result.output).toBe('The answer is 5.');
expect(result.toolCalls[0]).toMatchObject({ name: 'add', arguments: { a: 2, b: 3 } });
expect(mock.getCallCount()).toBe(2);

await cog.close();

The tool executes for real between the two scripted responses. For responses that depend on the request, implement LLMBackend yourself — see Your Own Scripted Backend.

A few things that matter in tests:

  • The runtime retries failed LLM calls (2 retries by default). Set llm: { backends, retry: false } when you script errors.
  • Streaming (chatStream) is used only when the run passes both stream: true and onToken.
  • Pass threadId to reuse a conversation; give the Cogitator a memory adapter (cog.memory = new MockMemoryAdapter()) to inspect what is saved.
  • Call cog.close() after each test to release memory adapters and sandboxes.

Workflow Tests

Workflows built only from function nodes never touch the Cogitator, so the workflows package's own tests pass a stub ({} as Cogitator). Use a mock backend as above when the workflow contains agentNodes.

import { WorkflowBuilder, WorkflowExecutor, functionNode } from '@cogitator-ai/workflows';
import type { Cogitator } from '@cogitator-ai/core';

type PipelineState = { data?: string; processed?: boolean };

const workflow = new WorkflowBuilder<PipelineState>('test-pipeline')
  .addNode(
    'fetch',
    functionNode<PipelineState>('fetch', async () => 'test', {
      stateMapper: (data) => ({ data: data as string }),
    })
  )
  .addNode(
    'process',
    functionNode<PipelineState>('process', async () => true, {
      stateMapper: (processed) => ({ processed: processed as boolean }),
    }),
    { after: ['fetch'] }
  )
  .build();

const executor = new WorkflowExecutor({} as Cogitator);
const result = await executor.execute(workflow);

expect(result.error).toBeUndefined();
expect(result.state).toMatchObject({ data: 'test', processed: true });

A failing node does not reject execute(): the result carries it in error.

End-to-End Tests

Unit tests live next to each package (packages/<pkg>/src/__tests__/*.test.ts); tests against real providers live in packages/e2e/src/__tests__/<pkg>/*.e2e.ts and need GOOGLE_API_KEY, OPENAI_API_KEY and/or a running Ollama. When a test uses Google, pass the key through providers:

const cog = new Cogitator({
  llm: {
    defaultModel: 'google/gemini-3.5-flash-lite',
    providers: { google: { apiKey: process.env.GOOGLE_API_KEY! } },
  },
});

Running Tests

pnpm test                                             # unit tests of every package
pnpm --filter @cogitator-ai/core test                 # one package
pnpm --filter @cogitator-ai/e2e test                  # all e2e tests
pnpm --filter @cogitator-ai/e2e test -- --run src/__tests__/core   # e2e tests of one package

Packages use Vitest; pnpm --filter <pkg> test:watch (where defined) or npx vitest inside a package starts watch mode.

Observability

Trace agent executions with Langfuse and OpenTelemetry — cost tracking, span visualization, and production debugging.

Test Utilities

@cogitator-ai/test-utils — a scriptable mock LLM backend, a recording memory adapter, fixtures and async/stream helpers for testing Cogitator code.

On this page

What to TestUnit Testing ToolsAgent Tests with a Mock BackendWorkflow TestsEnd-to-End TestsRunning Tests