# Evaluation Framework

URL: https://cogitator.app/docs/evals

> Systematically evaluate LLM agents with @cogitator-ai/evals — dataset-driven testing, deterministic and LLM-as-judge metrics, assertions, A/B comparison, and CI-ready reporters.

## Why Evaluate?

LLM outputs are non-deterministic. A prompt change that improves one response might degrade ten others. Without systematic evaluation, you're flying blind. `@cogitator-ai/evals` gives you a structured way to measure agent quality, catch regressions, and compare model configurations — all with statistical rigor.

## Installation

```bash
pnpm add @cogitator-ai/evals
```

Optional dependencies: `papaparse` for CSV datasets, and `@cogitator-ai/core` for agent targets and LLM-as-judge metrics:

```bash
pnpm add papaparse
pnpm add @cogitator-ai/core
```

## Quick Start

```typescript
import { Dataset, EvalSuite, exactMatch, contains, threshold } from '@cogitator-ai/evals';

const dataset = Dataset.from([
  { input: 'What is 2+2?', expected: '4' },
  { input: 'Capital of France?', expected: 'Paris' },
  { input: 'Largest planet?', expected: 'Jupiter' },
]);

const suite = new EvalSuite({
  dataset,
  target: {
    fn: async (input) => {
      // your agent or LLM call here
      return 'some response';
    },
  },
  metrics: [exactMatch(), contains()],
  assertions: [threshold('exactMatch', 0.8)],
});

const result = await suite.run();
result.report('console');
```

## Architecture

```
┌──────────┐    ┌───────────┐    ┌─────────┐    ┌────────────┐
│ Dataset  │───▶│ EvalSuite │───▶│ Metrics │───▶│ Assertions │
└──────────┘    └───────────┘    └─────────┘    └────────────┘
                      │                                │
                      ▼                                ▼
                ┌───────────┐                   ┌────────────┐
                │  Target   │                   │ Reporters  │
                └───────────┘                   └────────────┘
```

**Eval flow:** Load test cases from a Dataset (JSONL, CSV, or programmatic) → EvalSuite runs each case against a Target (agent or function) with concurrency control → Metrics score each response → Scores are aggregated (mean, median, p95, etc.) → Assertions check thresholds and regressions → Reporters output results to console, JSON, CSV, or CI.

## EvalSuite

The core class that orchestrates evaluation runs.

```typescript
import { EvalSuite, exactMatch, latency, threshold } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { fn: async (input) => answer(input) },
  metrics: [exactMatch()],
  statisticalMetrics: [latency()],
  assertions: [threshold('exactMatch', 0.8)],
  concurrency: 5,
  timeout: 30000,
  retries: 1,
  onProgress: ({ completed, total }) => console.log(`${completed}/${total}`),
});
```

| Option               | Default | Description                                                                                                                      |
| -------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `dataset`            | —       | [`Dataset`](/docs/evals/datasets) of cases                                                                                       |
| `target`             | —       | What to evaluate — see [Target](#target)                                                                                         |
| `metrics`            | `[]`    | Per-case [metrics](/docs/evals/metrics)                                                                                          |
| `statisticalMetrics` | `[]`    | Metrics over the whole run (`latency()`, `cost()`, `tokenUsage()`)                                                               |
| `judge`              | —       | `{ model, temperature?, maxTokens?, cogitator? }` (`JudgeConfigInput`; `temperature` defaults to `0`) — required for LLM metrics |
| `assertions`         | `[]`    | [Pass/fail checks](/docs/evals/assertions)                                                                                       |
| `concurrency`        | `5`     | Cases run in parallel                                                                                                            |
| `timeout`            | `30000` | Per-attempt timeout in ms (minimum 1000)                                                                                         |
| `retries`            | `0`     | Extra attempts for a failed or timed-out case (max 10)                                                                           |
| `onProgress`         | —       | Called with `{ completed, total, currentCase }` after each case                                                                  |

Errors and timeouts are retried up to `retries` times. If every attempt fails, the case result has an empty `output` and an `error` message (for example `Timed out after 30000ms`), so a failed run is never mistaken for an empty answer. Metrics still score that empty output.

### Target

The target defines what you're evaluating. Use `fn` for a simple function, or `agent` + `cogitator` for a full Cogitator agent:

```typescript
import { Agent, Cogitator } from '@cogitator-ai/core';
import type { EvalTarget } from '@cogitator-ai/evals';

const fnTarget: EvalTarget = {
  fn: async (input: string) => myApp.answer(input),
};

const cogitator = new Cogitator();
const agentTarget: EvalTarget = {
  agent: new Agent({
    name: 'support',
    model: 'openai/gpt-5.5',
    instructions: 'Answer support questions briefly.',
  }),
  cogitator,
};
```

A target has either `fn` or both `agent` and `cogitator`; anything else throws when the suite is created. An agent target runs `cogitator.run(agent, { input, context })` with the case's `context`, and records the run's `usage` (tokens, cost) and `toolCalls` on the case result — that is what the `cost()` and `tokenUsage()` metrics read. A `fn` target only records the duration.

### Results

`suite.run()` returns `EvalSuiteResult` with everything you need:

```typescript
const result = await suite.run();

result.results; // per-case { case, output, duration, error?, usage?, toolCalls?, scores }
result.aggregated; // { exactMatch: { mean, median, p95, ... }, ... }
result.assertions; // [{ name, passed, message }, ...]
result.stats; // { total, duration (ms), cost (sum of case usage.cost) }

result.report('console');
result.report(['json', 'csv'], { path: './eval-report' });
result.saveBaseline('./baseline.json');
```

Results keep the dataset order regardless of concurrency. See [Reporters](/docs/evals/reporters) and [baselines](/docs/evals/assertions#noregression).

To try a suite on a slice of the dataset, pass `EvalRunOptions`:

```typescript
const smoke = await suite.run({ maxCases: 5 });
```

Only the first `maxCases` cases run; `results`, `aggregated`, `assertions`, `stats.total` and the `onProgress` totals cover just those cases. `maxCases` must be a positive integer, otherwise `run()` throws.

## EvalBuilder

Fluent builder API as an alternative to direct `EvalSuite` construction:

```typescript
import {
  EvalBuilder,
  Dataset,
  exactMatch,
  contains,
  latency,
  threshold,
} from '@cogitator-ai/evals';

const suite = new EvalBuilder()
  .withDataset(dataset)
  .withTarget({ fn: myFn })
  .withMetrics([exactMatch(), contains()])
  .withStatisticalMetrics([latency()])
  .withAssertions([threshold('exactMatch', 0.9)])
  .withConcurrency(10)
  .withTimeout(15000)
  .withRetries(1)
  .onProgress(({ completed, total }) => console.log(`${completed}/${total}`))
  .build();

const result = await suite.run();
```

Use `.withJudge({ model, temperature?, cogitator })` when the metrics include LLM-as-judge metrics; `build()` throws without it, or without a dataset or target.

## Agent Tool

`evalTools(suite)` returns a `run_eval` tool so an agent can run a suite and read the summary:

```typescript
import { Agent, tool } from '@cogitator-ai/core';
import { evalTools } from '@cogitator-ai/evals';

const [runEval] = evalTools(suite);

const qaAgent = new Agent({
  name: 'qa',
  model: 'openai/gpt-5.5',
  instructions: 'Run the eval suite and report regressions.',
  tools: [tool(runEval)],
});
```

The tool takes an optional `maxCases` and returns `{ success, total, duration, cost, metrics, assertionsPassed }`, where `metrics` holds the mean of each metric. With `maxCases` the tool calls `suite.run({ maxCases })`, so only the first N cases run and `total`, `metrics` and `assertionsPassed` are computed from them. Errors come back as `{ success: false, error }`. `createRunEvalTool(suite)` builds the same tool.

## Next Steps

* [Datasets](/docs/evals/datasets) — load test cases from JSONL, CSV, or build programmatically
* [Metrics](/docs/evals/metrics) — deterministic, LLM-as-judge, statistical, and custom metrics
* [Assertions](/docs/evals/assertions) — threshold checks, regression detection, custom assertions
* [A/B Comparison](/docs/evals/comparison) — compare two targets with statistical significance testing
* [Reporters](/docs/evals/reporters) — console, JSON, CSV, and CI output formats