Context Management

OverviewCopy Anchor Link

Long-running agents accumulate conversation history that can exceed a model's context window. The ContextManager automatically detects when messages are approaching the limit and compresses them using one of four strategies -- keeping agents running indefinitely without losing critical context.

import { Cogitator } from '@cogitator-ai/core';

const cog = new Cogitator({
  llm: { providers: { openai: { apiKey: process.env.OPENAI_API_KEY! } } },
  context: {
    enabled: true,
    strategy: 'hybrid',
    compressionThreshold: 0.8,
    outputReserve: 0.15,
    windowSize: 10,
  },
});

Context management is built into the runtime: whenever context is set (and enabled is not false), the run checks the conversation before each model call and compresses it when needed. Compression changes only what is sent; memory keeps the full history. After compression, tool results whose call was dropped (and calls without their results) are removed too, so the provider never sees a broken tool exchange.

ConfigurationCopy Anchor Link

interface ContextManagerConfig {
  enabled?: boolean; // default: true
  strategy?: 'truncate' | 'sliding-window' | 'summarize' | 'hybrid';
  compressionThreshold?: number; // 0-1, triggers at this % of max tokens (default: 0.8)
  outputReserve?: number; // fraction of context reserved for output (default: 0.15)
  summaryModel?: string; // provider/model id for LLM summaries (see below)
  windowSize?: number; // messages to keep in sliding window (default: 10)
}

The compressionThreshold controls when compression kicks in. At 0.8, compression triggers when token usage exceeds 80% of the available context (the model's context window minus outputReserve). Compression then brings the conversation back under the available context.

Which strategies call a model depends on summaryModel: with it, sliding-window, summarize and hybrid compression summarize older messages with that model. Without it, only strategy: 'summarize' calls a model (the running agent's); sliding-window and hybrid fall back to an extractive summary without an LLM call.

Token CountingCopy Anchor Link

Cogitator uses a fast heuristic token counter (4 characters per token plus 4 per message) for real-time decisions. Context windows come from the model registry, with built-in fallbacks for well-known model families; an unknown model counts as 8,192 tokens:

import { ContextManager } from '@cogitator-ai/core';

const manager = new ContextManager({
  enabled: true,
  strategy: 'hybrid',
  compressionThreshold: 0.8,
});

manager.getModelContextLimit('anthropic/claude-sonnet-5-5'); // 1000000
manager.getModelContextLimit('ollama/my-finetune'); // 8192

Checking Context StateCopy Anchor Link

Before compressing, you can inspect the current state:

const state = manager.checkState(messages, 'openai/gpt-6.1-sol');

console.log(state.currentTokens); // estimated tokens used
console.log(state.maxTokens); // effective limit (minus output reserve)
console.log(state.availableTokens); // remaining space
console.log(state.utilizationPercent); // e.g. 85.3
console.log(state.needsCompression); // true if above threshold

StrategiesCopy Anchor Link

TruncateCopy Anchor Link

Drops the oldest non-system messages to fit within the token budget. Fast and stateless -- no LLM call required.

const cog = new Cogitator({
  context: { strategy: 'truncate', compressionThreshold: 0.8 },
  // ...
});

System messages are always preserved. History is trimmed from the beginning, keeping the most recent exchanges intact. Best for scenarios where older context is rarely needed.

Sliding WindowCopy Anchor Link

Keeps a fixed window of recent messages and optionally summarizes older ones. Provides a good balance between context preservation and token efficiency.

const cog = new Cogitator({
  context: {
    strategy: 'sliding-window',
    windowSize: 15,
    summaryModel: 'openai/gpt-6-luna',
  },
  // ...
});

Messages outside the last windowSize are summarized into a single system message ([Previous conversation summary]): by summaryModel when it is set, else as a basic extractive summary of the last few user and assistant messages. If even the window does not leave room for a summary, the older messages are dropped.

SummarizeCopy Anchor Link

Uses an LLM to generate a comprehensive summary of older conversation history, preserving key facts, decisions, and pending tasks.

const cog = new Cogitator({
  context: {
    strategy: 'summarize',
    summaryModel: 'openai/gpt-6-luna',
    windowSize: 5,
  },
  // ...
});

The summarizer keeps the most recent 20% of messages as is (at least 2, at most windowSize) and replaces everything older with one summary message. It uses summaryModel, or the running agent's model when that is not set. The summary prompt instructs the LLM to preserve user preferences, key decisions, and critical context. Falls back to an extractive summary if the LLM call fails.

Hybrid (Default)Copy Anchor Link

Automatically selects the best strategy based on how far over the limit the conversation is:

const cog = new Cogitator({
  context: { strategy: 'hybrid', summaryModel: 'openai/gpt-6-luna' },
  // ...
});

This is the recommended default -- it adapts to the situation without manual tuning.

Compression ResultsCopy Anchor Link

Every compression call returns detailed metrics:

const result = await manager.compress(messages, 'openai/gpt-6.1-sol');

console.log(result.messages); // compressed message array
console.log(result.originalTokens); // tokens before compression
console.log(result.compressedTokens); // tokens after compression
console.log(result.strategy); // which strategy was used
console.log(result.truncated); // messages dropped (truncate)
console.log(result.summarized); // messages summarized (summarize/sliding-window)

Standalone UsageCopy Anchor Link

You can use ContextManager outside of the Cogitator runtime for custom pipelines:

import { ContextManager } from '@cogitator-ai/core';

const manager = new ContextManager(
  {
    enabled: true,
    strategy: 'sliding-window',
    windowSize: 10,
    summaryModel: 'openai/gpt-6-luna',
  },
  { route: (model) => cog.route(model) }
);

if (manager.shouldCompress(messages, 'openai/gpt-6.1-sol')) {
  const { messages: compressed } = await manager.compress(messages, 'openai/gpt-6.1-sol');
  messages = compressed;
}

The second constructor argument tells the manager how to reach a model for summaries: route returns the backend and the model name for a provider/model id (cog.route() does exactly that), and getBackend is the older form that returns only the backend. Without either, summaries are extractive. The strategy classes (TruncateStrategy, SlidingWindowStrategy, SummarizeStrategy, HybridStrategy) are exported too, for custom pipelines that call compress() on a strategy directly.