Prompt Versions

OverviewCopy Anchor Link

An agent's instructions in code are its starting point. cogitator.prompts keeps versions on top of them: deploy a new version and every following run of that agent uses it; roll back and the old one is live again. Each version collects how its runs went — run count, success rate, average score, latency and cost — so you can tell whether a change helped.

const v2 = await cog.prompts.deploy(
  writer,
  'You write release notes. Lead with what changed for the user.'
);

const result = await cog.run(writer, { input }); // runs with v2
result.prompt; // { key: 'writer', versionId: v2.id, version: 2 }

await cog.prompts.current(writer); // v2
await cog.prompts.rollbackTo(writer); // back to the previous version (recorded as a new one)
await cog.prompts.rollbackTo(writer, v2.id); // or to a specific one
await cog.prompts.history(writer); // newest first, with metrics

Versions are kept per agent id when you set one, else per name, so they outlive the process when the stores are durable. deploy, history and the rest take the agent or that key. Before the first deploy, runs use the agent's own instructions. A run resumed after a pause keeps the instructions it started with.

A/B TestsCopy Anchor Link

Test new instructions against the current ones on real traffic. Each thread gets one variant for the length of the test, so a conversation never changes voice halfway:

await cog.prompts.startABTest(writer, {
  name: 'shorter notes',
  treatment: 'You write release notes in at most five bullet points.',
  treatmentAllocation: 0.3, // 30% of threads
  minSampleSize: 50,
});

const result = await cog.run(writer, { input, threadId });
result.prompt?.abTest; // { id, variant: 'control' | 'treatment' }

The control is the current version, or the agent's own instructions before any deploy. The variant is derived from the test and the threadId, so pass the conversation's threadId: a run without one gets a new thread, and with it an independent draw. activeABTest(writer) returns the running test.

Every run's score is recorded for its variant (not for the current version, whose metrics count only runs outside a test). Once the two variants together have twice minSampleSize runs (default 50 each) and the difference is significant (Welch's t-test at confidenceLevel, default 0.95), or maxDuration (default 7 days) has passed, the test completes; with autoDeployWinner a winning treatment becomes the next version. completeABTest(id) ends a test early and returns its outcome.

Scoring RunsCopy Anchor Link

By default a completed run scores 1 and a failed one 0, which compares success rates (a failed run's score is always 0). Score what matters to you instead — an eval metric, a user's thumbs-up stored by run id, the length of the answer:

const cog = new Cogitator({
  prompts: {
    score: async (result) => (await judge.evaluate(result.output)).score, // 0 – 1
    autoDeployWinner: true,
  },
});

StoresCopy Anchor Link

Versions and tests live in process memory unless you pass stores. PostgresTraceStore from the learning module holds both:

import { PostgresTraceStore } from '@cogitator-ai/core';

const store = new PostgresTraceStore({ connectionString: process.env.DATABASE_URL! });
await store.connect();

const cog = new Cogitator({
  prompts: { versions: store.instructionVersions(), abTests: store.abTests() },
});

Any object implementing InstructionVersionStore and ABTestStore works too.