Memory Benchmarks

The memory of a Cogitator run can be built five ways, from keeping only the newest turns to compacting the thread or extracting memory items from it. This page is the measurement behind the defaults: every strategy answered the same 2,222 questions of seven long-memory datasets on the same model and context budget, and its answers were scored the way each dataset scores them.

Read the numbers as a comparison of the strategies of this runtime, not as leaderboard figures. One model answered, ran the memory stages and judged, the context was held at 8192 tokens on purpose so memory has to choose, and the datasets were adapted to the runtime as described under Methodology.

ResultsCopy Anchor Link

Averaged over the seven datasets, each dataset counting once:

The answer cost counts the questions only, per question answered: the 600 failed MemoryAgentBench questions of extracted made no answer call. Compaction and extraction also pay once per conversation for their own model calls: $36.42 for compacting every conversation of the run and $31.60 for extracting from them, against $9.78 for all answers of all five strategies.

• relevant and hybrid beat recent on every dataset, by 7 to 67 points, and their intervals overlap on every dataset. hybrid costs a quarter less, keeps more of the prompt cache and has the lower tail latency, for about 50 ms more context build at the median.
• compacted wins up to about 100K tokens (BEAM 100K +19.1, LongMemEval-S +11.0) and is no better than recent past that: -2.5 points on BEAM 500K (interval -6.9 to +1.2), -2.6 on BEAM 1M and -7.6 on MemoryAgentBench. It sends 55% fewer input tokens per question.
• extracted gains 5 to 16 points over recent on BEAM and LongMemEval-S, most on preferences (BEAM 500K preference following 87.5% against 69.0%), and stays below relevant and hybrid on every dataset.

By datasetCopy Anchor Link

Accuracy is the mean score of the questions, a failed answer counted as zero. "vs recent" is the difference in points on the same questions. Both intervals are 95% cluster bootstrap intervals, see Scoring.

LongMemEval-SCopy Anchor Link

BEAM 100KCopy Anchor Link

BEAM 500KCopy Anchor Link

BEAM 1MCopy Anchor Link

MemoryAgentBench conflictsCopy Anchor Link

LongMemEval-V2 smallCopy Anchor Link

LoCoMo (smoke)Copy Anchor Link

What the numbers decidedCopy Anchor Link

A memory stage is on by default only where it won without a cost worth noticing. Everything else is an option, with the advice below.

The memory stages ran on claude-haiku-5-5, a cheap model, and it is the only one measured, so nothing here says how another model would do in them.

FailuresCopy Anchor Link

Failures are counted as zero in every accuracy and interval above. Answers that failed and passed when the run was resumed are not counted.

The extraction failures were a fault of the extractor, not of the model. A window of these conversations holds 130 to 270 numbered facts, and a model that lists them all writes 10K to 22K tokens against the 8000 an extraction call may write, so its answer was cut inside the JSON array. The extractor then read no item from it and asked again the same way, three times. The extractor now keeps the items a cut answer finished and asks the model to go on after the last one, and it reads the array past brackets in the text around it. The numbers on this page were measured before that fix, so extracted on MemoryAgentBench measures the failure, not extraction, and the two smaller conversations it answered (6K tokens each) are its only data there.

CaveatsCopy Anchor Link

• LoCoMo is a smoke test: 100 sampled questions of 10 dialogues, scored by token F1 against short reference answers, which a full sentence scores low. Its absolute numbers mean little, and its ordering of the strategies broadly agrees with the other datasets.
• LongMemEval-V2 small has two haystacks (web-1, enterprise-0) and MemoryAgentBench eight conversations, so their intervals, which resample conversations, are wide.
• One model answered, ran the stages and judged, and the judge graded the answers of the same model. The judges follow each benchmark's own prompts, not the model each benchmark's paper used.
• compacted and extracted answer on recent over what the stage left (the summary and the kept tail, or the memory items as facts), so they measure the stage alone, not the stage combined with retrieval.

MethodologyCopy Anchor Link

SetupCopy Anchor Link

DatasetsCopy Anchor Link

Every dataset is downloaded at a pinned revision and never committed.

The adaptations to the runtime are part of the method: every turn carries the date of its session, the dataset's answering rules and its "today" go to the agent's instructions so retrieval searches the question alone, MemoryAgentBench facts go in as messages of about 2048 tokens (the input limit of the embedding model), LongMemEval-V2 trajectories go in as notes of earlier sessions with accessibility trees cut to 4000 characters and its questions with screenshots left out, and BEAM event ordering is scored by its rubric judge without the Kendall tau alignment.

StrategiesCopy Anchor Link

ScoringCopy Anchor Link

Scores run from 0 to 1 per question (a BEAM rubric gives half points), and a question whose answer failed, or whose retrieval fell back to the newest turns, scores 0. The questions of one conversation share its history, so they are not independent: the 95% intervals are cluster bootstrap intervals, 1000 resamples of whole conversations, and the difference to recent is resampled the same way over the paired questions.

Offline suitesCopy Anchor Link

Two suites run on every benchmark run with an in-process model and cost nothing, and both passed in full:

• Consistency (36 checks, 12 per retrieval strategy): a history larger than the budget, forks and their branches, rewinds, forgetting and redacting turns and the memory items from them, and runs without errors or fallbacks.
• Compaction loss (16 checks): three rounds of compaction folded into arcs and a saga, every planted open item kept in the handoff and the levels, an item closed with evidence shown as done and one closed without it kept open, unverified quotes marked, the kept tail verbatim, junk answers asked again and never published.

Running itCopy Anchor Link

The benchmark lives in packages/gauntlet/src/memory-bench, with the prompts and evaluators it takes from the benchmarks and their licenses. A run resumes from its results, so nothing paid for is asked again:

pnpm memory-bench --estimate
pnpm memory-bench --datasets longmemeval-s,beam-100k,beam-500k,beam-1m,mab-conflict,lme-v2-small,locomo \
  --strategies recent,relevant,hybrid,compacted,extracted \
  --sample longmemeval-s=100,beam-500k=200,beam-1m=200,locomo=100 --run my-run
pnpm memory-bench --consistency-only