Assertions

OverviewCopy Anchor Link

Assertions run after all metrics are aggregated and produce pass/fail results. They're the quality gates that determine whether an eval run succeeded. Use them in CI to block deployments when quality drops.

interface AssertionResult {
  name: string;
  passed: boolean;
  message: string;
  actual?: number;
  expected?: number;
}

Assertions receive aggregated metric data — mean, median, min, max, stdDev and p50/p95/p99 for each metric, per-case and statistical alike — plus overall suite stats (total cases, duration, cost). Their results are in result.assertions; a failed assertion does not make suite.run() throw.

thresholdCopy Anchor Link

The most common assertion. Checks that a metric's mean score meets a minimum value. For latency and cost metrics (detected by name prefix), the check is inverted — the actual value must be at or below the threshold. A metric missing from the results fails the assertion.

import { EvalSuite, contains, cost, exactMatch, latency, threshold } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { agent, cogitator },
  metrics: [exactMatch(), contains()],
  statisticalMetrics: [latency(), cost()],
  assertions: [
    threshold('exactMatch', 0.9), // mean exactMatch >= 0.9
    threshold('contains', 0.95), // mean contains >= 0.95
    threshold('latency.p95', 5000), // p95 latency <= 5000ms
    threshold('cost.mean', 0.01), // mean cost <= $0.01
  ],
});

Dot-path AccessCopy Anchor Link

You can access nested aggregation fields with dot notation:

Direction DetectionCopy Anchor Link

Metrics whose names start with latency, cost or tokenUsage (or end with Duration or Latency) are treated as "lower is better" — the assertion passes when the actual value is at or below the threshold, so threshold('tokenUsage', 2000) caps the mean at 2000 tokens. All other metrics are "higher is better". threshold and noRegression use the same rule.

noRegressionCopy Anchor Link

Compares current results against a saved baseline file. Fails if any metric regresses beyond a tolerance percentage.

Step 1: Save a baselineCopy Anchor Link

Run your eval suite and save the baseline:

const result = await suite.run();
result.saveBaseline('./baseline.json');

This writes a JSON file mapping every aggregated metric (statistical ones included) to its mean:

{
  "exactMatch": 0.92,
  "contains": 0.97
}

Step 2: Assert no regressionCopy Anchor Link

In subsequent runs, compare against the baseline:

import { noRegression } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [exactMatch(), contains()],
  assertions: [
    noRegression('./baseline.json'),
    noRegression('./baseline.json', { tolerance: 0.1 }), // 10% tolerance
  ],
});

For "higher is better" metrics, the assertion fails if actual < baseline * (1 - tolerance). For "lower is better" metrics (latency, cost, tokenUsage), it fails if actual > baseline * (1 + tolerance). Only means are compared, and baseline metrics missing from the current run are skipped. The assertion reports the first regression it finds, and also fails when the baseline file cannot be read or none of its metrics are in the current results.

Baseline WorkflowCopy Anchor Link

A typical CI workflow:

1. Establish a baseline on main: run evals, saveBaseline('./baseline.json'), commit it
2. On each PR: run evals with noRegression('./baseline.json')
3. If the PR improves scores: update the baseline file
4. If the PR regresses: fix the issue or adjust tolerance

Custom AssertionsCopy Anchor Link

Build arbitrary pass/fail logic with the assertion() factory:

import { assertion } from '@cogitator-ai/evals';

const budgetCheck = assertion({
  name: 'budgetCheck',
  check: (_aggregated, stats) => stats.cost < 1.0,
  message: 'Total eval cost exceeded $1.00 budget',
});

const speedCheck = assertion({
  name: 'speedCheck',
  check: (aggregated) => {
    const latency = aggregated.latency;
    return latency ? latency.p95 < 3000 : true;
  },
  message: 'P95 latency exceeded 3 seconds',
});

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [exactMatch()],
  statisticalMetrics: [latency()],
  assertions: [budgetCheck, speedCheck],
});

The check function receives:

• aggregated — Record<string, AggregatedMetric> with per-metric stats
• stats — { total: number, duration: number, cost: number } for the full run

Return true for pass, false for fail. The optional message is shown on failure; a check that throws fails with the error message.

Combining AssertionsCopy Anchor Link

Pass multiple assertions; each produces its own result:

assertions: [
  threshold('exactMatch', 0.9),
  threshold('relevance', 0.8),
  noRegression('./baseline.json'),
  budgetCheck,
];

The report('ci') reporter exits with code 1 if any assertion fails, making it straightforward to integrate with CI pipelines. Outside CI, check result.assertions.every((a) => a.passed).