# Metrics

URL: https://cogitator.app/docs/evals/metrics

> Score agent responses with deterministic checks, LLM-as-judge evaluation, statistical aggregation, and custom metric functions.

## Overview

Metrics score individual eval case results. Each metric receives the case result — the case (`input`, `expected`, `context`, `metadata`), the `output`, `duration`, and for agent targets `usage` and `toolCalls` — and returns a score between 0 and 1. Scores are automatically aggregated across the dataset into mean, median, min, max, standard deviation and p50/p95/p99.

```typescript
interface MetricScore {
  name: string;
  score: number;
  details?: string;
  metadata?: Record<string, unknown>;
}
```

## Deterministic Metrics

Fast, zero-cost metrics that compare output against expected values using string operations or schema validation.

### exactMatch

Binary check — does the output exactly match the expected value? Trims whitespace from both sides. Cases without `expected` score 0 (as do `contains` cases).

```typescript
import { exactMatch } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [exactMatch()],
});
```

Case-sensitive mode:

```typescript
exactMatch({ caseSensitive: true });
```

| Option          | Type    | Default | Description              |
| --------------- | ------- | ------- | ------------------------ |
| `caseSensitive` | boolean | false   | Require exact case match |

### contains

Checks whether the output contains the expected string as a substring.

```typescript
import { contains } from '@cogitator-ai/evals';

contains(); // case-insensitive (default)
contains({ caseSensitive: true }); // case-sensitive
```

### regex

Tests the output against a regular expression pattern.

```typescript
import { regex } from '@cogitator-ai/evals';

regex(/\d{4}-\d{2}-\d{2}/); // matches date format
regex('^(yes|no)$'); // string pattern
```

### jsonSchema

Validates that the output is valid JSON conforming to a Zod schema. Useful for evaluating structured output agents.

```typescript
import { z } from 'zod';
import { jsonSchema } from '@cogitator-ai/evals';

const schema = z.object({
  answer: z.string(),
  confidence: z.number().min(0).max(1),
});

jsonSchema(schema);
```

Returns score 1 if the whole output parses as JSON and passes validation, 0 otherwise — output wrapped in Markdown code fences does not parse. The `details` field contains the parse or validation error on failure.

## LLM-as-Judge Metrics

Use an LLM to evaluate subjective qualities like faithfulness, relevance, and coherence. These metrics require a `judge` config in the EvalSuite (and `@cogitator-ai/core` installed). The judge runs as a Cogitator agent on `judge.model` (a `provider/model` id): pass the runtime as `judge.cogitator`, or use an agent target, whose `cogitator` is used when `judge.cogitator` is not set. A suite with LLM metrics and nothing to run the judge fails when it is built.

```typescript
import { Cogitator } from '@cogitator-ai/core';
import { EvalSuite, faithfulness, relevance, coherence, helpfulness } from '@cogitator-ai/evals';

const cogitator = new Cogitator({
  llm: { providers: { openai: { apiKey: process.env.OPENAI_API_KEY! } } },
});

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [faithfulness(), relevance(), coherence(), helpfulness()],
  judge: {
    model: 'openai/gpt-6.1-sol',
    temperature: 0,
    cogitator, // runs the judge; defaults to the target's cogitator for agent targets
  },
});
```

| Judge option  | Description                                                             |
| ------------- | ----------------------------------------------------------------------- |
| `model`       | Judge model, e.g. `'openai/gpt-6.1-sol'`                                |
| `temperature` | Judge temperature; optional, defaults to `0` for repeatable scores      |
| `maxTokens`   | Optional cap on the judge's reply                                       |
| `cogitator`   | Runtime that runs the judge; defaults to the agent target's `cogitator` |

The option type is `JudgeConfigInput`; the suite parses it into `JudgeConfig`, filling in the `temperature` default.

Each judge metric sends one prompt per case containing the rubric, the case `input`, its `context` when it has one (each key on its own line, objects as JSON), the `expected` value (or `N/A`) and the response, and asks for JSON with `score` and `reasoning`; the judge agent (one iteration, no memory) requests that shape with a JSON schema. The score is clamped to \[0, 1] and `reasoning` becomes the score's `details`. An answer without a parsable score, or a judge error, counts as 0 with the reason in `details` — it never fails the run.

### Built-in Judge Metrics

| Metric           | Evaluates                                                                   |
| ---------------- | --------------------------------------------------------------------------- |
| `faithfulness()` | Whether the response is faithful to facts in the input and the case context |
| `relevance()`    | Whether the response addresses the question asked                           |
| `coherence()`    | Whether the response is logical and well-structured                         |
| `helpfulness()`  | Whether the response would be useful to the user                            |

### Custom Judge Metrics

Create a judge metric with your own evaluation prompt:

```typescript
import { llmMetric } from '@cogitator-ai/evals';

const tone = llmMetric({
  name: 'professionalTone',
  prompt: 'Evaluate whether the response maintains a professional, business-appropriate tone.',
});

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [tone],
  judge: { model: 'openai/gpt-6.1-sol', temperature: 0, cogitator },
});
```

The framework appends scoring instructions to your prompt automatically.

### Manual Judge Binding

For advanced use cases — a judge that is not a Cogitator agent, for example — bind the judge context manually instead of passing `judge` to EvalSuite. The bound metric is an ordinary metric, so the suite needs no `judge` config:

```typescript
import { faithfulness, bindJudgeContext } from '@cogitator-ai/evals';

const bound = bindJudgeContext(faithfulness(), {
  cogitator: { run: async ({ input }) => ({ output: await callLLM(input) }) },
  judgeConfig: { model: 'my-judge', temperature: 0 },
});

const suite = new EvalSuite({ dataset, target: { fn }, metrics: [bound] });
```

Your `run` receives the full judge prompt as `input` and should return the model's text; `judgeConfig` is only carried along, so pick the model inside `run`. Without binding, a judge metric used outside a suite scores 0 (`unbound judge metric`).

## Statistical Metrics

Aggregate metrics that operate on the full result set rather than individual cases. They compute distributions over latency, cost, and token usage.

```typescript
import { latency, cost, tokenUsage } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { agent, cogitator },
  metrics: [exactMatch()],
  statisticalMetrics: [latency(), cost(), tokenUsage()],
});
```

Each statistical metric reports one value per case; `result.aggregated.<name>` holds their mean, median, min, max, standard deviation and p50/p95/p99, so thresholds such as `threshold('latency.p95', 5000)` check real numbers. Extra figures are in `metadata`.

### latency

Reports execution time distribution across all cases, in milliseconds.

```typescript
// result.aggregated.latency.metadata:
// { p50, p95, p99, mean, median, min, max }
```

### cost

Reports cost distribution in USD. Only cases with `usage` count — that is, agent targets (a `fn` target records no usage).

```typescript
// result.aggregated.cost.metadata:
// { total, mean, median, min, max, p95, p99 }
```

### tokenUsage

Reports token consumption per case (input plus output). Like `cost`, it only counts cases with `usage`. Assertions treat it as lower-is-better, so `threshold('tokenUsage', n)` caps the mean at `n` — see [Direction Detection](/docs/evals/assertions#direction-detection).

```typescript
// result.aggregated.tokenUsage.metadata:
// { totalInput, totalOutput, totalTokens, meanInput, meanOutput }
```

## Custom Metrics

Create a fully custom metric with the `metric()` factory:

```typescript
import { metric } from '@cogitator-ai/evals';

const wordCount = metric({
  name: 'wordCount',
  evaluate: ({ output, expected }) => {
    const actual = output.split(/\s+/).length;
    const target = expected ? parseInt(expected) : 100;
    const ratio = Math.min(actual, target) / Math.max(actual, target);
    return { score: ratio, details: `${actual} words (target: ${target})` };
  },
});

const responseLength = metric({
  name: 'responseLength',
  evaluate: async ({ output }) => {
    const len = output.length;
    const score = Math.min(len / 500, 1);
    return { score };
  },
});
```

The `evaluate` function receives `{ input, output, expected, context }` and must return `{ score, details? }`. The score is automatically clamped to \[0, 1]. Both sync and async evaluate functions are supported; if `evaluate` throws, the case scores 0 with the error in `details`.

For access to the whole case result — `duration`, `usage`, `toolCalls`, `error` — write a `MetricFn` directly. It must carry a `metricName`:

```typescript
import type { EvalCaseResult, MetricFn, MetricScore } from '@cogitator-ai/evals';

const usedSearch: MetricFn = Object.assign(
  async (result: EvalCaseResult): Promise<MetricScore> => ({
    name: 'usedSearch',
    score: result.toolCalls?.some((call) => call.name === 'search') ? 1 : 0,
  }),
  { metricName: 'usedSearch' }
);
```

## Combining Metrics

Pass any combination of metric types to EvalSuite:

```typescript
const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [exactMatch(), contains(), faithfulness(), wordCount],
  statisticalMetrics: [latency(), cost()],
  judge: { model: 'openai/gpt-6.1-sol', temperature: 0, cogitator },
});
```

Deterministic and LLM metrics run per-case. Statistical metrics run once over the full result set after all cases complete.