# A/B Comparison

URL: https://cogitator.app/docs/evals/comparison

> Compare two agent configurations with statistical significance testing using paired t-tests and McNemar's test.

## Overview

`EvalComparison` runs the same dataset against two targets (baseline and challenger) and determines which performs better — with statistical significance. It uses paired t-tests for continuous metrics and McNemar's test for binary metrics (0 or 1 scores).

Only per-case `metrics` are compared, and a higher mean is better. `statisticalMetrics` (latency, cost, tokens) are computed for each side and available in `result.baseline.aggregated` / `result.challenger.aggregated`, but they do not take part in the comparison.

```typescript
interface ComparisonResult {
  summary: {
    winner: 'baseline' | 'challenger' | 'tie';
    metrics: Record<string, MetricComparison>;
  };
  baseline: EvalSuiteResult;
  challenger: EvalSuiteResult;
}

interface MetricComparison {
  baseline: number; // mean score
  challenger: number; // mean score
  pValue: number; // statistical significance
  significant: boolean; // p < 0.05
  winner: 'baseline' | 'challenger' | 'tie';
}
```

## Basic Usage

```typescript
import { Dataset, EvalComparison, exactMatch, contains } from '@cogitator-ai/evals';

const dataset = Dataset.from([
  { input: 'What is 2+2?', expected: '4' },
  { input: 'Capital of France?', expected: 'Paris' },
  { input: 'Largest ocean?', expected: 'Pacific' },
  // ... more cases for statistical power
]);

const comparison = new EvalComparison({
  dataset,
  targets: {
    baseline: {
      fn: async (input) => callModel('gpt-6-luna', input),
    },
    challenger: {
      fn: async (input) => callModel('gpt-6.1-sol', input),
    },
  },
  metrics: [exactMatch(), contains()],
});

const result = await comparison.run();
```

## Reading Results

```typescript
const { summary, baseline, challenger } = result;

console.log(`Winner: ${summary.winner}`);

for (const [metric, comp] of Object.entries(summary.metrics)) {
  console.log(
    `${metric}: baseline=${comp.baseline.toFixed(3)} ` +
      `challenger=${comp.challenger.toFixed(3)} ` +
      `p=${comp.pValue.toFixed(4)} ` +
      `significant=${comp.significant} ` +
      `winner=${comp.winner}`
  );
}

// full suite results are available too
baseline.report('console');
challenger.report('console');
```

## Statistical Tests

The comparison automatically selects the right test based on the data:

### Paired t-Test

Used for continuous scores (e.g., faithfulness: 0.73, 0.85, 0.91). Tests whether the mean difference between paired observations is significantly different from zero.

### McNemar's Test

Used for binary scores (all values are exactly 0 or 1, e.g., exactMatch). Tests whether the proportion of disagreements between baseline and challenger is significantly different from random.

Both tests use p \< 0.05 as the significance threshold. A metric's winner is only declared when the difference is statistically significant, and is then the side with the higher mean. With fewer than 2 cases every metric is a tie (`pValue: 1`). Scores are paired by case; a case where a metric is missing counts as 0.

The tests are exported for your own analyses:

```typescript
import { pairedTTest, mcnemarsTest, aggregate } from '@cogitator-ai/evals';

const t = pairedTTest([0.7, 0.8, 0.9], [0.75, 0.9, 0.95]);
// { tStatistic, degreesOfFreedom, pValue, significant, confidenceInterval }

const m = mcnemarsTest(3, 12); // A right & B wrong, A wrong & B right
// { chiSquare, pValue, significant }

const stats = aggregate([120, 340, 95]);
// { mean, median, min, max, stdDev, p50, p95, p99 }
```

`mean`, `median`, `stdDev` and `percentile(values, p)` (with `p` from 0 to 1) are exported as well.

## Winner Determination

The overall winner is determined by majority vote across metrics:

1. Each metric independently determines its winner (or tie) based on statistical significance
2. Count wins for baseline and challenger across all metrics
3. If challenger has more wins: `'challenger'`
4. If baseline has more wins: `'baseline'`
5. If equal: `'tie'`

## Full Example

```typescript
import { Cogitator } from '@cogitator-ai/core';
import {
  Dataset,
  EvalComparison,
  exactMatch,
  contains,
  faithfulness,
  latency,
} from '@cogitator-ai/evals';

const dataset = await Dataset.fromJsonl('./data/eval-cases.jsonl');
const cogitator = new Cogitator({
  llm: { providers: { openai: { apiKey: process.env.OPENAI_API_KEY! } } },
});

const comparison = new EvalComparison({
  dataset,
  targets: {
    baseline: {
      fn: async (input) => {
        const res = await fetch('http://localhost:3000/v1/chat', {
          method: 'POST',
          body: JSON.stringify({ model: 'gpt-6-luna', input }),
        });
        const json = (await res.json()) as { output: string };
        return json.output;
      },
    },
    challenger: {
      fn: async (input) => {
        const res = await fetch('http://localhost:3000/v1/chat', {
          method: 'POST',
          body: JSON.stringify({ model: 'gpt-6.1-sol', input }),
        });
        const json = (await res.json()) as { output: string };
        return json.output;
      },
    },
  },
  metrics: [exactMatch(), contains(), faithfulness()],
  statisticalMetrics: [latency()],
  judge: { model: 'openai/gpt-6.1-sol', temperature: 0, cogitator },
  concurrency: 10,
  timeout: 30000,
  onProgress: ({ target, completed, total }) => {
    console.log(`[${target}] ${completed}/${total}`);
  },
});

const result = await comparison.run();

console.log(`\nOverall winner: ${result.summary.winner}`);
for (const [name, mc] of Object.entries(result.summary.metrics)) {
  const sig = mc.significant ? '***' : '';
  console.log(
    `  ${name}: ${mc.baseline.toFixed(3)} vs ${mc.challenger.toFixed(3)} ` +
      `(p=${mc.pValue.toFixed(4)}) ${mc.winner}${sig}`
  );
}
```

## Options

| Option               | Type                   | Default | Description                                                                                      |
| -------------------- | ---------------------- | ------- | ------------------------------------------------------------------------------------------------ |
| `dataset`            | Dataset                | —       | Test cases to run                                                                                |
| `targets.baseline`   | EvalTarget             | —       | Baseline target                                                                                  |
| `targets.challenger` | EvalTarget             | —       | Challenger target                                                                                |
| `metrics`            | MetricFn\[]            | —       | Per-case metrics                                                                                 |
| `statisticalMetrics` | StatisticalMetricFn\[] | —       | Aggregate metrics                                                                                |
| `judge`              | JudgeConfigInput       | —       | Required for LLM metrics (`temperature` defaults to `0`); with `fn` targets it needs `cogitator` |
| `concurrency`        | number                 | 5       | Parallel case execution per target                                                               |
| `timeout`            | number                 | 30000   | Per-case timeout in ms                                                                           |
| `retries`            | number                 | 0       | Retry count for failed cases                                                                     |
| `onProgress`         | function               | —       | Progress callback with target label                                                              |

## Tips for Meaningful Comparisons

* **Use enough data.** Statistical tests need sample sizes to detect real differences. 30+ cases is a minimum; 100+ is better.
* **Control variables.** Change one thing at a time — model, prompt, temperature — so you know what caused the difference.
* **Check p-values.** A metric showing `significant: false` means the observed difference could be due to random variation.
* **Run both targets concurrently.** `EvalComparison` runs baseline and challenger in parallel, so external factors (API latency, rate limits) affect both equally.