# Assertions

URL: https://cogitator.app/docs/evals/assertions

> Set pass/fail criteria with threshold checks, baseline regression detection, and custom assertion logic.

## Overview

Assertions run after all metrics are aggregated and produce pass/fail results. They're the quality gates that determine whether an eval run succeeded. Use them in CI to block deployments when quality drops.

```typescript
interface AssertionResult {
  name: string;
  passed: boolean;
  message: string;
  actual?: number;
  expected?: number;
}
```

Assertions receive aggregated metric data — mean, median, min, max, stdDev and p50/p95/p99 for each metric, per-case and statistical alike — plus overall suite stats (total cases, duration, cost). Their results are in `result.assertions`; a failed assertion does not make `suite.run()` throw.

## threshold

The most common assertion. Checks that a metric's mean score meets a minimum value. For latency and cost metrics (detected by name prefix), the check is inverted — the actual value must be at or below the threshold. A metric missing from the results fails the assertion.

```typescript
import { EvalSuite, contains, cost, exactMatch, latency, threshold } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { agent, cogitator },
  metrics: [exactMatch(), contains()],
  statisticalMetrics: [latency(), cost()],
  assertions: [
    threshold('exactMatch', 0.9), // mean exactMatch >= 0.9
    threshold('contains', 0.95), // mean contains >= 0.95
    threshold('latency.p95', 5000), // p95 latency <= 5000ms
    threshold('cost.mean', 0.01), // mean cost <= $0.01
  ],
});
```

### Dot-path Access

You can access nested aggregation fields with dot notation:

| Path             | Resolves to                  |
| ---------------- | ---------------------------- |
| `exactMatch`     | `aggregated.exactMatch.mean` |
| `exactMatch.p95` | `aggregated.exactMatch.p95`  |
| `latency.p99`    | `aggregated.latency.p99`     |
| `cost.median`    | `aggregated.cost.median`     |

### Direction Detection

Metrics whose names start with `latency`, `cost` or `tokenUsage` (or end with `Duration` or `Latency`) are treated as "lower is better" — the assertion passes when the actual value is at or below the threshold, so `threshold('tokenUsage', 2000)` caps the mean at 2000 tokens. All other metrics are "higher is better". `threshold` and `noRegression` use the same rule.

## noRegression

Compares current results against a saved baseline file. Fails if any metric regresses beyond a tolerance percentage.

### Step 1: Save a baseline

Run your eval suite and save the baseline:

```typescript
const result = await suite.run();
result.saveBaseline('./baseline.json');
```

This writes a JSON file mapping every aggregated metric (statistical ones included) to its mean:

```json
{
  "exactMatch": 0.92,
  "contains": 0.97
}
```

### Step 2: Assert no regression

In subsequent runs, compare against the baseline:

```typescript
import { noRegression } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [exactMatch(), contains()],
  assertions: [
    noRegression('./baseline.json'),
    noRegression('./baseline.json', { tolerance: 0.1 }), // 10% tolerance
  ],
});
```

| Option      | Type   | Default | Description                                  |
| ----------- | ------ | ------- | -------------------------------------------- |
| `tolerance` | number | 0.05    | Allowed regression as a fraction (0.05 = 5%) |

For "higher is better" metrics, the assertion fails if `actual < baseline * (1 - tolerance)`. For "lower is better" metrics (latency, cost, tokenUsage), it fails if `actual > baseline * (1 + tolerance)`. Only means are compared, and baseline metrics missing from the current run are skipped. The assertion reports the first regression it finds, and also fails when the baseline file cannot be read or none of its metrics are in the current results.

### Baseline Workflow

A typical CI workflow:

1. Establish a baseline on `main`: run evals, `saveBaseline('./baseline.json')`, commit it
2. On each PR: run evals with `noRegression('./baseline.json')`
3. If the PR improves scores: update the baseline file
4. If the PR regresses: fix the issue or adjust tolerance

## Custom Assertions

Build arbitrary pass/fail logic with the `assertion()` factory:

```typescript
import { assertion } from '@cogitator-ai/evals';

const budgetCheck = assertion({
  name: 'budgetCheck',
  check: (_aggregated, stats) => stats.cost < 1.0,
  message: 'Total eval cost exceeded $1.00 budget',
});

const speedCheck = assertion({
  name: 'speedCheck',
  check: (aggregated) => {
    const latency = aggregated.latency;
    return latency ? latency.p95 < 3000 : true;
  },
  message: 'P95 latency exceeded 3 seconds',
});

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [exactMatch()],
  statisticalMetrics: [latency()],
  assertions: [budgetCheck, speedCheck],
});
```

The `check` function receives:

* `aggregated` — `Record<string, AggregatedMetric>` with per-metric stats
* `stats` — `{ total: number, duration: number, cost: number }` for the full run

Return `true` for pass, `false` for fail. The optional `message` is shown on failure; a `check` that throws fails with the error message.

## Combining Assertions

Pass multiple assertions; each produces its own result:

```typescript
assertions: [
  threshold('exactMatch', 0.9),
  threshold('relevance', 0.8),
  noRegression('./baseline.json'),
  budgetCheck,
];
```

The `report('ci')` reporter exits with code 1 if any assertion fails, making it straightforward to integrate with CI pipelines. Outside CI, check `result.assertions.every((a) => a.passed)`.