Plugin Evals

OverviewCopy Anchor Link

A plugin is only as good as what it makes the agent do. Plugin evals measure that: a suite of cases, each a realistic prompt plus graders, runs several times with the plugin loaded and several times without it, and every run is scored. The difference between the two scores, the delta, is what the plugin contributed.

The suite format is the one of Claude Code's claude plugin eval, so a suite written for Claude Code runs on Cogitator and the other way round. Run it from the command line with cogitator plugin eval, or from code with experimental_runPluginEvals() of @cogitator-ai/evals. Both are experimental.

cogitator plugin eval .

CASE                  WITH  W/OUT  Δ      RUNS  COST   NOTES
writes-commit-message 1.00  0.33   +0.67  6     $0.41

1 case(s) · mean Δ +0.67 · 74s · $0.41
Report: /Users/you/my-plugin/evals/results/2026-10-27T17-02-11-482Z/report.html

Every run and every judge vote is a real model call, counted against your provider's bill.

Suite LayoutCopy Anchor Link

A suite lives in evals/ inside the plugin. A directory with a prompt.md or a case.yaml is a case, a directory without one groups cases, and everything inside a case directory belongs to it.

my-plugin/
├── .claude-plugin/plugin.json
├── skills/commit-message/SKILL.md
└── evals/
    ├── writes-commit-message/
    │   ├── prompt.md              # frontmatter: run settings, body: the prompt
    │   ├── case.yaml              # optional: context fields, or the whole case
    │   └── graders/
    │       ├── criteria.md        # frontmatter: type and options, body: rubric or pattern
    │       └── skill-fired.md
    ├── mocks/
    │   └── tracker/
    │       ├── create_issue.md    # one mocked MCP tool
    │       ├── _tools.json        # optional: the real tool schemas
    │       └── fixtures/
    └── results/<timestamp>/       # written by each run, add results/ to .gitignore
        ├── aggregate-result.json
        └── report.html

To keep the suite elsewhere, set "experimental": { "evals": "quality/evals" } in plugin.json or pass --eval-dir quality/evals, the flag winning. The path must be relative and made of plain directory names, a leading ./ is fine, and in plugin.json a list of paths takes its first entry. An absolute path or one with .. is refused: as a flag value it is an error, in the manifest it prints a Warning: line and evals/ is used.

prompt.mdCopy Anchor Link

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
tags: [smoke]
---

Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.

The body is the prompt, sent as it is. The frontmatter takes these fields, and an unknown key is an error:

case.yamlCopy Anchor Link

case.yaml describes a case in YAML, alone or next to prompt.md. It needs schema_version: "1.1" and name. description, tags, plugins, runs and expected_outcome go at the top level, the run fields of prompt.md under execution:. When both files exist, the prompt.md frontmatter wins field by field, its body is the prompt, and graders/*.md come after the graders: of case.yaml.

schema_version: '1.1'
name: changelog-from-diff
tags: [smoke]
context:
  scaffold_script: fixture.sh
  add_dirs: [resources]
graders:
  - name: changelog-written
    type: file_exists
    path: CHANGELOG.md

The scaffold script runs with a small fixed environment (PATH, a temporary HOME and TMPDIR, TERM=dumb) and 120 seconds at most. If it fails, the run scores 0 with a scaffold failed error. A case with a history_file runs one arm by default, since the recorded conversation already reflects the plugin.

GradersCopy Anchor Link

Each file under graders/ is one check applied after a run, named after the file. Every grader takes a type, a weight (default 1, any positive number) and an optional arm.

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?commit-message"'
---

target of a regex grader and focus of an llm grader take the same values:

Tool calls are graded under Claude Code's names and input shapes: Read, Write, Edit, Glob, Grep, Bash, Skill, Agent, TodoWrite, WebFetch, WebSearch and mcp__<server>__<tool>. A grader may also name the Cogitator tool, such as run_command. read_skill_file, which reads a file of an active skill, keeps its own name: only use_skill calls count as Skill.

Two Arms and the DeltaCopy Anchor Link

A case normally runs in two arms: the with-arm with the plugin loaded and the without-arm with no plugin. A run's score is the weighted share of its graders that passed, a case's score the mean of its with-arm runs, and its delta the with-arm score minus the without-arm score. A case passes when its score is at least --threshold, 1.0 by default. The delta is reported but never changes the exit code.

A case runs the with-arm only under --ablation none, or when it resumes a transcript and --ablation with-without was not given. A single-arm case has no delta.

A check like "the skill was invoked" can never pass without the plugin, so counting it would only inflate the delta. In a two-arm run these graders are kept out of the score in both arms and reported in the with-arm as plugin-fired indicators (scored: false):

• Every tool_used grader whose tool is Skill
• Every regex grader with target: mock_calls and every llm grader with focus: mock_calls, when each mocked server of the case is one the plugin declares
• Every grader with arm: with-only

When every grader of a case would be excluded, they are scored normally. arm: both scores a grader in both arms anyway, for a "must not invoke the skill" check with min: 0 and max: 0. Under --ablation none nothing is excluded.

ToolsCopy Anchor Link

Runs never stop to ask. A run gets the read-only tools its case lists in allowed_tools (Read, Glob, Grep, NotebookRead, Skill, AskUserQuestion, Agent, TodoWrite, TaskCreate, TaskGet, TaskList, TaskUpdate, TaskStop) plus what --allow-tools grants for every case, and nothing else: a tool that is not granted is not given to the model. A case that lists a tool the run did not grant shows it as not granted in the progress output.

cogitator plugin eval . --allow-tools Write Edit "Bash(npm test *)"

A grant with a specifier, such as Bash(npm test *) or WebFetch(domain:example.com), gives the tool and refuses every call the rule does not allow, with the reason the model reads. A skill's allowed-tools and a case's allowed_tools cannot widen the grants. Each run gets plugin data and settings of its own: the plugins' ${CLAUDE_PLUGIN_DATA} lives in the run's temporary folder, ${CLAUDE_PROJECT_DIR} is its workspace, and no settings file is read. The grants hold for the agents a run delegates to through Agent as well, and their calls are recorded in the run's tool calls with the delegated agent's name in agent, so the graders see them.

MocksCopy Anchor Link

A plugin whose skills call MCP tools is evaluated without the real service. One Markdown file per tool goes under evals/mocks/<server>/<tool>.md for the whole suite, or under a case's own mocks/ for one case (overriding the suite's file by file), where <server> is the server's name in the plugin's MCP configuration. The plugin's real MCP servers never start. A mocked tool is named mcp__plugin_<plugin>_<server>__<tool> and needs no grant, a tool without a mock file is not available.

---
expect:
  title: string
  priority: [low, medium, high]
---

Created issue #4821: {{input.title}}

The body is the tool's answer, with {{input.<field>}} for a field of the call and {{file:fixtures/{input.id}.json}} for a file beside the mock. expect: maps dotted input paths to a type name (string, number, boolean, array, object, null), a /regex/ of a small dialect (literals, ., escapes, classes, the quantifiers * + ? {m,n}, an optional ^ and $, the flags i and s), a literal or a list of literals. A call that breaks it aborts the run with score 0 and an aborted record of the server, the tool and the reason. error: true returns the body as a tool error. _tools.json beside the mocks, a saved tools/list response, gives the mocked tools their real descriptions and input schemas.

Running EvalsCopy Anchor Link

cogitator plugin eval [target] [options]

The target is a plugin directory (default .), a single prompt.md or case.yaml, or an installed plugin as name or name@marketplace, whose results go under ./evals/results/ of the working directory. Put the target before --tag, --allow-tools and --json, which read the words after them as their values. The models and their keys come from the project's cogitator.yml and the environment, as for cogitator run.

A progress line prints as each run finishes, with its score, cost, duration and each grader's verdict. With one arm for every case the table shows SCORE and PASS% instead of WITH, W/OUT and Δ. NOTES shows the highest-weight failing grader's explanation, or the run's error, from the with-arm.

Exit CodesCopy Anchor Link

When every case passed and the command still exits 1, a line on stderr names the cause, even under --json.

In CICopy Anchor Link

cogitator plugin eval . \
  --json results.json \
  --threshold 0.8 \
  --model anthropic/claude-sonnet-5-5 \
  --judge-model anthropic/claude-haiku-4-5 \
  --max-cost-usd 20

Pin both models so scores stay comparable, use --ablation none where the delta is not needed, and leave partial results and runs with skippedPaidGraders out of trend charts.

From CodeCopy Anchor Link

import { experimental_runPluginEvals } from '@cogitator-ai/evals';

const { result, reportFile } = await experimental_runPluginEvals({
  target: './my-plugin',
  llm: { defaultModel: 'anthropic/claude-sonnet-5-5' },
  threshold: 0.8,
  onProgress: (event) => {
    if (event.type === 'run') console.log(event.case, event.run.arm, event.run.score);
  },
});

The options are the flags in camelCase (caseGlob for --case, tags for --tag), plus llm (the LLM configuration of the runs and the judge), exec (how granted Bash commands are spawned, such as a launch that adds an OS sandbox), signal and onProgress. A suite that cannot start throws PluginEvalError.

The JSON ResultCopy Anchor Link

aggregate-result.json, and the --json output, is a versioned document for CI scripts. Field names are camelCase and new fields are added without renaming the ones there.

The document also carries the run configuration and every grader definition.

The HTML ReportCopy Anchor Link

report.html is one self-contained file with no scripts and no external requests, to attach to a CI job or open from disk. At the top, a verdict line ("Plugin effect: +33.3 pts vs baseline, improved 2, flat 1, regressed 0 of 3 cases") and tiles for the suite score, the ablation delta, the baseline score, the cases passing the threshold and the perfect runs. Each case card shows its delta and score bar with the threshold marked, with a red left edge when the delta is negative. Inside a case the with-plugin runs come first, then the baseline runs, each grader with a pass or fail chip: a failed grader is open with its explanation, an llm grader shows the judge's votes and what it was shown, and graders kept out of the score carry a plugin-fired indicator badge. The prompt and the grader definitions follow the runs.

Differences from Claude CodeCopy Anchor Link

• The agent under test is a Cogitator agent with workspace tools, not a claude -p session, so the same plugin can score differently on the two runtimes.
• type: agent mocks, _server.md and abort_when are not supported: a case that uses the server, through its own mocks/ or a plugin that declares it, fails to load and its error names the file, the other cases run. There are no agent-mock recordings to replay.
• The plugin's real MCP servers never start: --mocks record is accepted as the only mode, --mocks off and --allow-real-servers exit 1 with that reason.
• There is no trust prompt, and reports are never published: --trust-plugin, --no-publish and --publish-report are accepted and change nothing, so Claude Code's CI command line runs as it is.
• Command-line errors (an unknown option, a wrong value) exit 1, since 2 means a partial run.
• Skills-directory plugins (name@skills-dir) and plugins with a plugin.json at the root instead of .claude-plugin/plugin.json are not recognized.
• Granted Bash commands run as you in the run's workspace directory without an OS sandbox, unless the exec.launch option of experimental_runPluginEvals() wraps them in one. Prefer suites you wrote, or a container.
• The llm judge reads text only: a binary file, an image included, fails the grader with a message to write it out as text.
• env variables reach the run's commands only, not the plugin's hooks.
• Claude Code tools without a Cogitator counterpart (NotebookRead, AskUserQuestion, TaskCreate, TaskGet, TaskList, TaskUpdate, MultiEdit) are not given, and the case's progress line says so.
• The judge is each case's agent model at temperature 1 unless --judge-model names another, where Claude Code uses its background model.
• max_turns counts model turns: the tool calls of the turn that reaches the limit do not run.
• The result names the runtime in cogitatorVersion instead of claudeVersion, and adds outcome, apiErrors and loadErrors.