Metrics

OverviewCopy Anchor Link

Metrics score individual eval case results. Each metric receives the case result — the case (input, expected, context, metadata), the output, duration, and for agent targets usage and toolCalls — and returns a score between 0 and 1. Scores are automatically aggregated across the dataset into mean, median, min, max, standard deviation and p50/p95/p99.

interface MetricScore {
  name: string;
  score: number;
  details?: string;
  metadata?: Record<string, unknown>;
}

Deterministic MetricsCopy Anchor Link

Fast, zero-cost metrics that compare output against expected values using string operations or schema validation.

exactMatchCopy Anchor Link

Binary check — does the output exactly match the expected value? Trims whitespace from both sides. Cases without expected score 0 (as do contains cases).

import { exactMatch } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [exactMatch()],
});

Case-sensitive mode:

exactMatch({ caseSensitive: true });

containsCopy Anchor Link

Checks whether the output contains the expected string as a substring.

import { contains } from '@cogitator-ai/evals';

contains(); // case-insensitive (default)
contains({ caseSensitive: true }); // case-sensitive

regexCopy Anchor Link

Tests the output against a regular expression pattern.

import { regex } from '@cogitator-ai/evals';

regex(/\d{4}-\d{2}-\d{2}/); // matches date format
regex('^(yes|no)$'); // string pattern

jsonSchemaCopy Anchor Link

Validates that the output is valid JSON conforming to a Zod schema. Useful for evaluating structured output agents.

import { z } from 'zod';
import { jsonSchema } from '@cogitator-ai/evals';

const schema = z.object({
  answer: z.string(),
  confidence: z.number().min(0).max(1),
});

jsonSchema(schema);

Returns score 1 if the whole output parses as JSON and passes validation, 0 otherwise — output wrapped in Markdown code fences does not parse. The details field contains the parse or validation error on failure.

LLM-as-Judge MetricsCopy Anchor Link

Use an LLM to evaluate subjective qualities like faithfulness, relevance, and coherence. These metrics require a judge config in the EvalSuite (and @cogitator-ai/core installed). The judge runs as a Cogitator agent on judge.model (a provider/model id): pass the runtime as judge.cogitator, or use an agent target, whose cogitator is used when judge.cogitator is not set. A suite with LLM metrics and nothing to run the judge fails when it is built.

import { Cogitator } from '@cogitator-ai/core';
import { EvalSuite, faithfulness, relevance, coherence, helpfulness } from '@cogitator-ai/evals';

const cogitator = new Cogitator({
  llm: { providers: { openai: { apiKey: process.env.OPENAI_API_KEY! } } },
});

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [faithfulness(), relevance(), coherence(), helpfulness()],
  judge: {
    model: 'openai/gpt-6.1-sol',
    temperature: 0,
    cogitator, // runs the judge; defaults to the target's cogitator for agent targets
  },
});

The option type is JudgeConfigInput; the suite parses it into JudgeConfig, filling in the temperature default.

Each judge metric sends one prompt per case containing the rubric, the case input, its context when it has one (each key on its own line, objects as JSON), the expected value (or N/A) and the response, and asks for JSON with score and reasoning; the judge agent (one iteration, no memory) requests that shape with a JSON schema. The score is clamped to [0, 1] and reasoning becomes the score's details. An answer without a parsable score, or a judge error, counts as 0 with the reason in details — it never fails the run.

Built-in Judge MetricsCopy Anchor Link

Custom Judge MetricsCopy Anchor Link

Create a judge metric with your own evaluation prompt:

import { llmMetric } from '@cogitator-ai/evals';

const tone = llmMetric({
  name: 'professionalTone',
  prompt: 'Evaluate whether the response maintains a professional, business-appropriate tone.',
});

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [tone],
  judge: { model: 'openai/gpt-6.1-sol', temperature: 0, cogitator },
});

The framework appends scoring instructions to your prompt automatically.

Manual Judge BindingCopy Anchor Link

For advanced use cases — a judge that is not a Cogitator agent, for example — bind the judge context manually instead of passing judge to EvalSuite. The bound metric is an ordinary metric, so the suite needs no judge config:

import { faithfulness, bindJudgeContext } from '@cogitator-ai/evals';

const bound = bindJudgeContext(faithfulness(), {
  cogitator: { run: async ({ input }) => ({ output: await callLLM(input) }) },
  judgeConfig: { model: 'my-judge', temperature: 0 },
});

const suite = new EvalSuite({ dataset, target: { fn }, metrics: [bound] });

Your run receives the full judge prompt as input and should return the model's text; judgeConfig is only carried along, so pick the model inside run. Without binding, a judge metric used outside a suite scores 0 (unbound judge metric).

Statistical MetricsCopy Anchor Link

Aggregate metrics that operate on the full result set rather than individual cases. They compute distributions over latency, cost, and token usage.

import { latency, cost, tokenUsage } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { agent, cogitator },
  metrics: [exactMatch()],
  statisticalMetrics: [latency(), cost(), tokenUsage()],
});

Each statistical metric reports one value per case; result.aggregated.<name> holds their mean, median, min, max, standard deviation and p50/p95/p99, so thresholds such as threshold('latency.p95', 5000) check real numbers. Extra figures are in metadata.

latencyCopy Anchor Link

Reports execution time distribution across all cases, in milliseconds.

// result.aggregated.latency.metadata:
// { p50, p95, p99, mean, median, min, max }

costCopy Anchor Link

Reports cost distribution in USD. Only cases with usage count — that is, agent targets (a fn target records no usage).

// result.aggregated.cost.metadata:
// { total, mean, median, min, max, p95, p99 }

tokenUsageCopy Anchor Link

Reports token consumption per case (input plus output). Like cost, it only counts cases with usage. Assertions treat it as lower-is-better, so threshold('tokenUsage', n) caps the mean at n — see Direction Detection.

// result.aggregated.tokenUsage.metadata:
// { totalInput, totalOutput, totalTokens, meanInput, meanOutput }

Custom MetricsCopy Anchor Link

Create a fully custom metric with the metric() factory:

import { metric } from '@cogitator-ai/evals';

const wordCount = metric({
  name: 'wordCount',
  evaluate: ({ output, expected }) => {
    const actual = output.split(/\s+/).length;
    const target = expected ? parseInt(expected) : 100;
    const ratio = Math.min(actual, target) / Math.max(actual, target);
    return { score: ratio, details: `${actual} words (target: ${target})` };
  },
});

const responseLength = metric({
  name: 'responseLength',
  evaluate: async ({ output }) => {
    const len = output.length;
    const score = Math.min(len / 500, 1);
    return { score };
  },
});

The evaluate function receives { input, output, expected, context } and must return { score, details? }. The score is automatically clamped to [0, 1]. Both sync and async evaluate functions are supported; if evaluate throws, the case scores 0 with the error in details.

For access to the whole case result — duration, usage, toolCalls, error — write a MetricFn directly. It must carry a metricName:

import type { EvalCaseResult, MetricFn, MetricScore } from '@cogitator-ai/evals';

const usedSearch: MetricFn = Object.assign(
  async (result: EvalCaseResult): Promise<MetricScore> => ({
    name: 'usedSearch',
    score: result.toolCalls?.some((call) => call.name === 'search') ? 1 : 0,
  }),
  { metricName: 'usedSearch' }
);

Combining MetricsCopy Anchor Link

Pass any combination of metric types to EvalSuite:

const suite = new EvalSuite({
  dataset,
  target: { fn },
  metrics: [exactMatch(), contains(), faithfulness(), wordCount],
  statisticalMetrics: [latency(), cost()],
  judge: { model: 'openai/gpt-6.1-sol', temperature: 0, cogitator },
});

Deterministic and LLM metrics run per-case. Statistical metrics run once over the full result set after all cases complete.