LLM as Judge

medium10 minllm-judge.ts

Free-text answers cannot be matched exactly. You want a model to grade faithfulness and relevance.

What it shows

Evaluate an agent with faithfulness() and relevance() and a judge model
Save a baseline

The code

llm-judge.ts

import { Agent, Cogitator } from '@cogitator-ai/core';
import { Dataset, EvalSuite, faithfulness, relevance } from '@cogitator-ai/evals';

const apiKey = process.env.GOOGLE_API_KEY;
if (!apiKey) throw new Error('Set GOOGLE_API_KEY');

const dataset = Dataset.from([
  {
    input: 'Explain what a closure is in JavaScript.',
    expected: 'A closure is a function that keeps access to variables from its lexical scope after the outer function returned.',
  },
  {
    input: 'What is the difference between let and var?',
    expected: 'let is block-scoped with a temporal dead zone; var is function-scoped and hoisted.',
  },
]);

const cog = new Cogitator({ llm: { providers: { google: { apiKey } } } });

const tutor = new Agent({
  name: 'js-tutor',
  model: 'google/gemini-3.5-flash-lite',
  instructions: 'You are a JavaScript tutor. Answer in 2-3 sentences.',
  temperature: 0.3,
  maxIterations: 1,
});

const suite = new EvalSuite({
  dataset,
  target: { agent: tutor, cogitator: cog },
  metrics: [faithfulness(), relevance()],
  judge: { model: 'google/gemini-3.5-flash-lite', temperature: 0 },
  concurrency: 1,
  onProgress: (progress) => console.log(`${progress.completed}/${progress.total}`),
});

const result = await suite.run();
result.report('console');
result.saveBaseline('evals-baseline.json');

await cog.close();

Run it

install · bash

pnpm add @cogitator-ai/core @cogitator-ai/evals

EnvironmentGOOGLE_API_KEY

run · bash

GOOGLE_API_KEY=your-key npx tsx llm-judge.ts

Or run the full example from a clone of the repository (keys in .env):

repository · bash

npx tsx examples/evals/02-llm-judge.ts

LLM as Judge

What it shows

The code

Run it

Learn more