Cogitator
Evaluation Framework

A/B Comparison

Compare two agent configurations with statistical significance testing using paired t-tests and McNemar's test.

Overview

EvalComparison runs the same dataset against two targets (baseline and challenger) and determines which performs better — with statistical significance. It uses paired t-tests for continuous metrics and McNemar's test for binary metrics (0 or 1 scores).

Only per-case metrics are compared, and a higher mean is better. statisticalMetrics (latency, cost, tokens) are computed for each side and available in result.baseline.aggregated / result.challenger.aggregated, but they do not take part in the comparison.

interface ComparisonResult {
  summary: {
    winner: 'baseline' | 'challenger' | 'tie';
    metrics: Record<string, MetricComparison>;
  };
  baseline: EvalSuiteResult;
  challenger: EvalSuiteResult;
}

interface MetricComparison {
  baseline: number; // mean score
  challenger: number; // mean score
  pValue: number; // statistical significance
  significant: boolean; // p < 0.05
  winner: 'baseline' | 'challenger' | 'tie';
}

Basic Usage

import { Dataset, EvalComparison, exactMatch, contains } from '@cogitator-ai/evals';

const dataset = Dataset.from([
  { input: 'What is 2+2?', expected: '4' },
  { input: 'Capital of France?', expected: 'Paris' },
  { input: 'Largest ocean?', expected: 'Pacific' },
  // ... more cases for statistical power
]);

const comparison = new EvalComparison({
  dataset,
  targets: {
    baseline: {
      fn: async (input) => callModel('gpt-6-luna', input),
    },
    challenger: {
      fn: async (input) => callModel('gpt-6.1-sol', input),
    },
  },
  metrics: [exactMatch(), contains()],
});

const result = await comparison.run();

Reading Results

const { summary, baseline, challenger } = result;

console.log(`Winner: ${summary.winner}`);

for (const [metric, comp] of Object.entries(summary.metrics)) {
  console.log(
    `${metric}: baseline=${comp.baseline.toFixed(3)} ` +
      `challenger=${comp.challenger.toFixed(3)} ` +
      `p=${comp.pValue.toFixed(4)} ` +
      `significant=${comp.significant} ` +
      `winner=${comp.winner}`
  );
}

// full suite results are available too
baseline.report('console');
challenger.report('console');

Statistical Tests

The comparison automatically selects the right test based on the data:

Paired t-Test

Used for continuous scores (e.g., faithfulness: 0.73, 0.85, 0.91). Tests whether the mean difference between paired observations is significantly different from zero.

McNemar's Test

Used for binary scores (all values are exactly 0 or 1, e.g., exactMatch). Tests whether the proportion of disagreements between baseline and challenger is significantly different from random.

Both tests use p < 0.05 as the significance threshold. A metric's winner is only declared when the difference is statistically significant, and is then the side with the higher mean. With fewer than 2 cases every metric is a tie (pValue: 1). Scores are paired by case; a case where a metric is missing counts as 0.

The tests are exported for your own analyses:

import { pairedTTest, mcnemarsTest, aggregate } from '@cogitator-ai/evals';

const t = pairedTTest([0.7, 0.8, 0.9], [0.75, 0.9, 0.95]);
// { tStatistic, degreesOfFreedom, pValue, significant, confidenceInterval }

const m = mcnemarsTest(3, 12); // A right & B wrong, A wrong & B right
// { chiSquare, pValue, significant }

const stats = aggregate([120, 340, 95]);
// { mean, median, min, max, stdDev, p50, p95, p99 }

mean, median, stdDev and percentile(values, p) (with p from 0 to 1) are exported as well.

Winner Determination

The overall winner is determined by majority vote across metrics:

  1. Each metric independently determines its winner (or tie) based on statistical significance
  2. Count wins for baseline and challenger across all metrics
  3. If challenger has more wins: 'challenger'
  4. If baseline has more wins: 'baseline'
  5. If equal: 'tie'

Full Example

import { Cogitator } from '@cogitator-ai/core';
import {
  Dataset,
  EvalComparison,
  exactMatch,
  contains,
  faithfulness,
  latency,
} from '@cogitator-ai/evals';

const dataset = await Dataset.fromJsonl('./data/eval-cases.jsonl');
const cogitator = new Cogitator({
  llm: { providers: { openai: { apiKey: process.env.OPENAI_API_KEY! } } },
});

const comparison = new EvalComparison({
  dataset,
  targets: {
    baseline: {
      fn: async (input) => {
        const res = await fetch('http://localhost:3000/v1/chat', {
          method: 'POST',
          body: JSON.stringify({ model: 'gpt-6-luna', input }),
        });
        const json = (await res.json()) as { output: string };
        return json.output;
      },
    },
    challenger: {
      fn: async (input) => {
        const res = await fetch('http://localhost:3000/v1/chat', {
          method: 'POST',
          body: JSON.stringify({ model: 'gpt-6.1-sol', input }),
        });
        const json = (await res.json()) as { output: string };
        return json.output;
      },
    },
  },
  metrics: [exactMatch(), contains(), faithfulness()],
  statisticalMetrics: [latency()],
  judge: { model: 'openai/gpt-6.1-sol', temperature: 0, cogitator },
  concurrency: 10,
  timeout: 30000,
  onProgress: ({ target, completed, total }) => {
    console.log(`[${target}] ${completed}/${total}`);
  },
});

const result = await comparison.run();

console.log(`\nOverall winner: ${result.summary.winner}`);
for (const [name, mc] of Object.entries(result.summary.metrics)) {
  const sig = mc.significant ? '***' : '';
  console.log(
    `  ${name}: ${mc.baseline.toFixed(3)} vs ${mc.challenger.toFixed(3)} ` +
      `(p=${mc.pValue.toFixed(4)}) ${mc.winner}${sig}`
  );
}

Options

OptionTypeDefaultDescription
datasetDataset—Test cases to run
targets.baselineEvalTarget—Baseline target
targets.challengerEvalTarget—Challenger target
metricsMetricFn[]—Per-case metrics
statisticalMetricsStatisticalMetricFn[]—Aggregate metrics
judgeJudgeConfigInput—Required for LLM metrics (temperature defaults to 0); with fn targets it needs cogitator
concurrencynumber5Parallel case execution per target
timeoutnumber30000Per-case timeout in ms
retriesnumber0Retry count for failed cases
onProgressfunction—Progress callback with target label

Tips for Meaningful Comparisons

  • Use enough data. Statistical tests need sample sizes to detect real differences. 30+ cases is a minimum; 100+ is better.
  • Control variables. Change one thing at a time — model, prompt, temperature — so you know what caused the difference.
  • Check p-values. A metric showing significant: false means the observed difference could be due to random variation.
  • Run both targets concurrently. EvalComparison runs baseline and challenger in parallel, so external factors (API latency, rate limits) affect both equally.

Assertions

Set pass/fail criteria with threshold checks, baseline regression detection, and custom assertion logic.

Reporters

Output evaluation results to the console, JSON files, CSV files, or CI-friendly format with automatic failure exit codes.

On this page

OverviewBasic UsageReading ResultsStatistical TestsPaired t-TestMcNemar's TestWinner DeterminationFull ExampleOptionsTips for Meaningful Comparisons