Cogitator
Evaluation Framework

Evaluation Framework

Systematically evaluate LLM agents with @cogitator-ai/evals — dataset-driven testing, deterministic and LLM-as-judge metrics, assertions, A/B comparison, and CI-ready reporters.

Why Evaluate?

LLM outputs are non-deterministic. A prompt change that improves one response might degrade ten others. Without systematic evaluation, you're flying blind. @cogitator-ai/evals gives you a structured way to measure agent quality, catch regressions, and compare model configurations — all with statistical rigor.

Installation

pnpm add @cogitator-ai/evals

Optional dependencies: papaparse for CSV datasets, and @cogitator-ai/core for agent targets and LLM-as-judge metrics:

pnpm add papaparse
pnpm add @cogitator-ai/core

Quick Start

import { Dataset, EvalSuite, exactMatch, contains, threshold } from '@cogitator-ai/evals';

const dataset = Dataset.from([
  { input: 'What is 2+2?', expected: '4' },
  { input: 'Capital of France?', expected: 'Paris' },
  { input: 'Largest planet?', expected: 'Jupiter' },
]);

const suite = new EvalSuite({
  dataset,
  target: {
    fn: async (input) => {
      // your agent or LLM call here
      return 'some response';
    },
  },
  metrics: [exactMatch(), contains()],
  assertions: [threshold('exactMatch', 0.8)],
});

const result = await suite.run();
result.report('console');

Architecture

┌──────────┐    ┌───────────┐    ┌─────────┐    ┌────────────┐
│ Dataset  │───▶│ EvalSuite │───▶│ Metrics │───▶│ Assertions │
└──────────┘    └───────────┘    └─────────┘    └────────────┘
                      │                                │
                      ▼                                ▼
                ┌───────────┐                   ┌────────────┐
                │  Target   │                   │ Reporters  │
                └───────────┘                   └────────────┘

Eval flow: Load test cases from a Dataset (JSONL, CSV, or programmatic) → EvalSuite runs each case against a Target (agent or function) with concurrency control → Metrics score each response → Scores are aggregated (mean, median, p95, etc.) → Assertions check thresholds and regressions → Reporters output results to console, JSON, CSV, or CI.

EvalSuite

The core class that orchestrates evaluation runs.

import { EvalSuite, exactMatch, latency, threshold } from '@cogitator-ai/evals';

const suite = new EvalSuite({
  dataset,
  target: { fn: async (input) => answer(input) },
  metrics: [exactMatch()],
  statisticalMetrics: [latency()],
  assertions: [threshold('exactMatch', 0.8)],
  concurrency: 5,
  timeout: 30000,
  retries: 1,
  onProgress: ({ completed, total }) => console.log(`${completed}/${total}`),
});
OptionDefaultDescription
dataset—Dataset of cases
target—What to evaluate — see Target
metrics[]Per-case metrics
statisticalMetrics[]Metrics over the whole run (latency(), cost(), tokenUsage())
judge—{ model, temperature?, maxTokens?, cogitator? } (JudgeConfigInput; temperature defaults to 0) — required for LLM metrics
assertions[]Pass/fail checks
concurrency5Cases run in parallel
timeout30000Per-attempt timeout in ms (minimum 1000)
retries0Extra attempts for a failed or timed-out case (max 10)
onProgress—Called with { completed, total, currentCase } after each case

Errors and timeouts are retried up to retries times. If every attempt fails, the case result has an empty output and an error message (for example Timed out after 30000ms), so a failed run is never mistaken for an empty answer. Metrics still score that empty output.

Target

The target defines what you're evaluating. Use fn for a simple function, or agent + cogitator for a full Cogitator agent:

import { Agent, Cogitator } from '@cogitator-ai/core';
import type { EvalTarget } from '@cogitator-ai/evals';

const fnTarget: EvalTarget = {
  fn: async (input: string) => myApp.answer(input),
};

const cogitator = new Cogitator();
const agentTarget: EvalTarget = {
  agent: new Agent({
    name: 'support',
    model: 'openai/gpt-5.5',
    instructions: 'Answer support questions briefly.',
  }),
  cogitator,
};

A target has either fn or both agent and cogitator; anything else throws when the suite is created. An agent target runs cogitator.run(agent, { input, context }) with the case's context, and records the run's usage (tokens, cost) and toolCalls on the case result — that is what the cost() and tokenUsage() metrics read. A fn target only records the duration.

Results

suite.run() returns EvalSuiteResult with everything you need:

const result = await suite.run();

result.results; // per-case { case, output, duration, error?, usage?, toolCalls?, scores }
result.aggregated; // { exactMatch: { mean, median, p95, ... }, ... }
result.assertions; // [{ name, passed, message }, ...]
result.stats; // { total, duration (ms), cost (sum of case usage.cost) }

result.report('console');
result.report(['json', 'csv'], { path: './eval-report' });
result.saveBaseline('./baseline.json');

Results keep the dataset order regardless of concurrency. See Reporters and baselines.

To try a suite on a slice of the dataset, pass EvalRunOptions:

const smoke = await suite.run({ maxCases: 5 });

Only the first maxCases cases run; results, aggregated, assertions, stats.total and the onProgress totals cover just those cases. maxCases must be a positive integer, otherwise run() throws.

EvalBuilder

Fluent builder API as an alternative to direct EvalSuite construction:

import {
  EvalBuilder,
  Dataset,
  exactMatch,
  contains,
  latency,
  threshold,
} from '@cogitator-ai/evals';

const suite = new EvalBuilder()
  .withDataset(dataset)
  .withTarget({ fn: myFn })
  .withMetrics([exactMatch(), contains()])
  .withStatisticalMetrics([latency()])
  .withAssertions([threshold('exactMatch', 0.9)])
  .withConcurrency(10)
  .withTimeout(15000)
  .withRetries(1)
  .onProgress(({ completed, total }) => console.log(`${completed}/${total}`))
  .build();

const result = await suite.run();

Use .withJudge({ model, temperature?, cogitator }) when the metrics include LLM-as-judge metrics; build() throws without it, or without a dataset or target.

Agent Tool

evalTools(suite) returns a run_eval tool so an agent can run a suite and read the summary:

import { Agent, tool } from '@cogitator-ai/core';
import { evalTools } from '@cogitator-ai/evals';

const [runEval] = evalTools(suite);

const qaAgent = new Agent({
  name: 'qa',
  model: 'openai/gpt-5.5',
  instructions: 'Run the eval suite and report regressions.',
  tools: [tool(runEval)],
});

The tool takes an optional maxCases and returns { success, total, duration, cost, metrics, assertionsPassed }, where metrics holds the mean of each metric. With maxCases the tool calls suite.run({ maxCases }), so only the first N cases run and total, metrics and assertionsPassed are computed from them. Errors come back as { success: false, error }. createRunEvalTool(suite) builds the same tool.

Next Steps

  • Datasets — load test cases from JSONL, CSV, or build programmatically
  • Metrics — deterministic, LLM-as-judge, statistical, and custom metrics
  • Assertions — threshold checks, regression detection, custom assertions
  • A/B Comparison — compare two targets with statistical significance testing
  • Reporters — console, JSON, CSV, and CI output formats

Realtime Mode

Native speech-to-speech with RealtimeSession — connect to OpenAI Realtime API or Gemini Live API for low-latency voice conversations with tool calling support.

Datasets

Load evaluation test cases from JSONL files, CSV files, or build them programmatically with filtering, sampling, and shuffling.

On this page

Why Evaluate?InstallationQuick StartArchitectureEvalSuiteTargetResultsEvalBuilderAgent ToolNext Steps