Tool Caching

OverviewCopy Anchor Link

The withCache() function wraps any tool with a caching layer. Repeated calls with the same (or semantically similar) parameters return cached results instantly, saving API calls, database queries, and compute time.

import { Agent, withCache, webSearch } from '@cogitator-ai/core';

const cachedSearch = withCache(webSearch, {
  strategy: 'exact',
  ttl: '10m',
  maxSize: 500,
  storage: 'memory',
});

const agent = new Agent({
  name: 'assistant',
  model: 'openai/gpt-6.1-sol',
  instructions: 'Answer questions using web search.',
  tools: [cachedSearch],
});

The cached tool is a drop-in replacement -- it has the same name, description, parameters and approval settings as the original, plus a .cache object for management. Only its execute changes, so caching works the same inside agent runs and when you call the tool directly.

Whatever execute returns is cached, including error objects such as { error: 'API key not found…' } that many built-in tools return instead of throwing. A thrown error is never cached. If a tool reports transient failures as data, wrap it so those results throw, or keep the TTL short.

withCache() APICopy Anchor Link

function withCache<TParams, TResult>(
  tool: Tool<TParams, TResult>,
  config: WithCacheOptions
): CachedTool<TParams, TResult>;

ConfigurationCopy Anchor Link

interface WithCacheOptions {
  strategy: 'exact' | 'semantic';
  ttl: DurationString;
  maxSize: number;
  storage: 'memory' | 'redis';
  similarity?: number;
  keyPrefix?: string;
  embeddingService?: EmbeddingService;
  redisClient?: RedisClientLike;
  onHit?: (key: string, params: unknown) => void;
  onMiss?: (key: string, params: unknown) => void;
  onEvict?: (key: string) => void;
}

maxSize caps the number of entries per storage; when it is reached, the least recently used entry is dropped. Expired entries are dropped when they are read.

Duration StringsCopy Anchor Link

TTL values use human-readable duration strings:

Cache StrategiesCopy Anchor Link

Exact MatchingCopy Anchor Link

Generates a deterministic cache key from the key prefix, the tool name and a SHA-256 hash of the parameters (serialized with sorted keys, so property order does not matter). Two calls with identical parameters hit the same cache entry.

const cachedHash = withCache(hash, {
  strategy: 'exact',
  ttl: '1h',
  maxSize: 1000,
  storage: 'memory',
});

Semantic MatchingCopy Anchor Link

Uses embedding vectors to find cached results for semantically similar inputs. Requires an embeddingService (any EmbeddingService from @cogitator-ai/memory, such as OpenAIEmbeddingService, or your own { embed, embedBatch, dimensions, model }). Without one, semantic behaves like exact.

const cachedSearch = withCache(webSearch, {
  strategy: 'semantic',
  ttl: '30m',
  maxSize: 200,
  storage: 'memory',
  similarity: 0.92,
  embeddingService: myEmbeddingService,
});

With semantic matching, a query like "TypeScript best practices" might match a cached result for "best practices for TypeScript development" if their cosine similarity exceeds the threshold.

The text that gets embedded is the first string among the parameters query, q, search, text, input, prompt and question, or the JSON of all parameters when there is none. A semantic hit therefore ignores the other parameters: { query: 'react hooks', maxResults: 20 } can be answered from the entry for { query: 'React hooks', maxResults: 5 }. Use exact when the other parameters change the result.

The lookup flow:

1. Check for an exact key match first
2. If no exact match, embed the current parameters and search for similar entries above the similarity threshold
3. If a similar entry is found, return its cached result
4. Otherwise, execute the tool and cache the result with its embedding

Embedding failures are ignored: the lookup falls back to running the tool, and the entry is stored without an embedding.

Storage BackendsCopy Anchor Link

In-Memory (LRU)Copy Anchor Link

Entries are stored in a Map with LRU eviction -- when maxSize is reached, the least recently accessed entry is evicted. Each cached tool gets its own storage.

const cached = withCache(sqlQuery, {
  strategy: 'exact',
  ttl: '5m',
  maxSize: 500,
  storage: 'memory',
});

In-memory storage is fast and requires no external dependencies, but entries are lost when the process restarts.

RedisCopy Anchor Link

For persistent, shared caching across multiple processes or server instances. Uses Redis sorted sets for LRU tracking and key-value storage for entries.

import Redis from 'ioredis';
import { withCache, webScrape } from '@cogitator-ai/core';

const redis = new Redis('redis://localhost:6379');

const cached = withCache(webScrape, {
  strategy: 'exact',
  ttl: '1h',
  maxSize: 5000,
  storage: 'redis',
  redisClient: redis,
  keyPrefix: 'myapp:tools',
});

An ioredis client matches RedisClientLike as it is, and so does a client from @cogitator-ai/redis:

import { createRedisClient } from '@cogitator-ai/redis';

const client = await createRedisClient({ url: 'redis://localhost:6379', keyPrefix: 'myapp:' });

const cachedScrape = withCache(webScrape, {
  strategy: 'exact',
  ttl: '1h',
  maxSize: 5000,
  storage: 'redis',
  redisClient: client,
});

Its scan is relative to the client's keyPrefix and walks every master in cluster mode, so cache.clear() and semantic lookups work in both modes; in a cluster, give the client a {hashtag} prefix so the storage's multi-key commands (mget, del) stay in one slot. Other clients need a small adapter: node-redis, for example, spells the commands setEx/zAdd and takes SCAN options as an object. Entries are written with SETEX, so Redis expires them on its own as well.

The storage keeps its data under keyPrefix (a : is appended when it does not end with one): each entry at <prefix>:entry:<cache key>, the LRU order in the sorted set <prefix>:lru and the entry count in <prefix>:counter. The cache key itself is <keyPrefix>:<toolName>:<hash> (again with a single : after the prefix), so with keyPrefix: 'myapp:tools' (the ioredis example) an entry lives at myapp:tools:entry:myapp:tools:web_scrape:<hash>. Cached tools that share a keyPrefix share one LRU and one maxSize, and cache.clear() deletes every key under the prefix. A keyPrefix set on the Redis client itself comes in front of all of these.

The RedisClientLike interface requires the methods used by Redis-backed cache storage:

interface RedisClientLike {
  get(key: string): Promise<string | null>;
  setex(key: string, seconds: number, value: string): Promise<string>;
  del(...keys: string[]): Promise<number>;
  mget(...keys: string[]): Promise<(string | null)[]>;
  zadd(key: string, score: number, member: string): Promise<number>;
  zrange(key: string, start: number, stop: string): Promise<string[]>;
  zrem(key: string, ...members: string[]): Promise<number>;
  incr(key: string): Promise<number>;
  decr(key: string): Promise<number>;
  exists(...keys: string[]): Promise<number>;
  scan(
    cursor: number | string,
    matchToken: 'MATCH',
    pattern: string,
    countToken: 'COUNT',
    count: number | string
  ): Promise<[cursor: string, keys: string[]]>;
}

You can also create storage instances directly:

import { createToolCacheStorage } from '@cogitator-ai/core';

const memoryStorage = createToolCacheStorage('memory', { maxSize: 1000 });

const redisStorage = createToolCacheStorage('redis', {
  redisClient: redis,
  keyPrefix: 'cache:',
  maxSize: 5000,
  onEvict: (key) => console.log('evicted', key),
});

createToolCacheStorage(type, { maxSize?, redisClient?, keyPrefix?, onEvict? }) returns a ToolCacheStorage (InMemoryToolCacheStorage or RedisToolCacheStorage, both exported; maxSize defaults to 1000, the Redis key prefix to toolcache:). onEvict is called with the key of each entry evicted to make room for a new one. withCache creates its own storage from its options, so use these when you build caching around something other than a tool.

Cache ManagementCopy Anchor Link

Every cached tool exposes a .cache object for runtime management:

StatsCopy Anchor Link

const stats = cachedSearch.cache.stats();
// {
//   hits: 142,
//   misses: 38,
//   size: 38,
//   evictions: 12,
//   hitRate: 0.789,
// }

ClearCopy Anchor Link

await cachedSearch.cache.clear();

hits, misses and hitRate count calls of this cached tool since it was created; evictions counts entries dropped to stay under maxSize.

InvalidateCopy Anchor Link

Remove a specific entry by its parameters (it resolves to whether an entry existed, and calls onEvict):

const existed = await cachedSearch.cache.invalidate({
  query: 'TypeScript tutorials',
  maxResults: 5,
});

WarmupCopy Anchor Link

Pre-populate the cache with known parameter-result pairs:

await cachedSearch.cache.warmup([
  {
    params: { query: 'React hooks guide', maxResults: 5 },
    result: { query: 'React hooks guide', provider: 'tavily', results: [] },
  },
  {
    params: { query: 'Node.js streams', maxResults: 5 },
    result: { query: 'Node.js streams', provider: 'tavily', results: [] },
  },
]);

When to CacheCopy Anchor Link

Cache tools that:

• Call external APIs (web search, scraping) -- save quota and reduce latency
• Run expensive queries (SQL, vector search) -- avoid duplicate database load
• Perform deterministic computation (hashing, math) -- same input always gives same output
• Have rate limits -- cache prevents hitting API rate limits on repeated queries

Avoid caching tools that:

• Have side effects (file_write, send_email, exec) -- caching would silently skip the action
• Must return real-time data (datetime, random_number) -- cached values would be stale
• Depend on external state that changes frequently between calls

Observability CallbacksCopy Anchor Link

Use the onHit, onMiss, and onEvict callbacks to integrate with your monitoring. onEvict receives the cache key of each entry removed by cache.invalidate() or dropped to stay under maxSize (in memory and in Redis); expired entries are dropped silently.

const cached = withCache(webSearch, {
  strategy: 'exact',
  ttl: '10m',
  maxSize: 500,
  storage: 'memory',
  onHit: (key, params) => {
    metrics.increment('tool_cache.hit', { tool: 'web_search' });
  },
  onMiss: (key, params) => {
    metrics.increment('tool_cache.miss', { tool: 'web_search' });
  },
  onEvict: (key) => {
    metrics.increment('tool_cache.evict', { tool: 'web_search' });
  },
});

HelpersCopy Anchor Link