Basic Eval Suite

easy5 minbasic-eval.ts

You want a test that fails CI when answer quality drops below a threshold.

What it shows

Build a Dataset, score with exactMatch and contains
Fail on threshold assertions

The code

basic-eval.ts

import { Dataset, EvalSuite, contains, exactMatch, threshold } from '@cogitator-ai/evals';

const capitals: Record<string, string> = {
  'What is the capital of France?': 'Paris',
  'What is the capital of Japan?': 'Tokyo',
  'What is the capital of Spain?': 'Madrid',
};

const dataset = Dataset.from(
  Object.entries(capitals).map(([input, expected]) => ({ input, expected }))
);

const suite = new EvalSuite({
  dataset,
  target: { fn: async (input: string) => capitals[input] ?? 'unknown' },
  metrics: [exactMatch(), contains()],
  assertions: [threshold('exactMatch', 0.8), threshold('contains', 0.9)],
});

const result = await suite.run();
result.report('console');

for (const caseResult of result.results) {
  console.log(caseResult.case.input, '->', caseResult.output, caseResult.scores.map((s) => `${s.name}=${s.score}`));
}

process.exitCode = result.assertions.every((assertion) => assertion.passed) ? 0 : 1;

Run it

install · bash

pnpm add @cogitator-ai/evals

run · bash

npx tsx basic-eval.ts

Or run the full example from a clone of the repository (keys in .env):

repository · bash

npx tsx examples/evals/01-basic-eval.ts

Basic Eval Suite

What it shows

The code

Run it

Learn more