# Tool Caching

URL: https://cogitator.app/docs/tools/tool-caching

> Cache tool results with TTL, LRU eviction, and semantic similarity matching using in-memory or Redis storage.

## Overview

The `withCache()` function wraps any tool with a caching layer. Repeated calls with the same (or semantically similar) parameters return cached results instantly, saving API calls, database queries, and compute time.

```typescript
import { Agent, withCache, webSearch } from '@cogitator-ai/core';

const cachedSearch = withCache(webSearch, {
  strategy: 'exact',
  ttl: '10m',
  maxSize: 500,
  storage: 'memory',
});

const agent = new Agent({
  name: 'assistant',
  model: 'openai/gpt-6.1-sol',
  instructions: 'Answer questions using web search.',
  tools: [cachedSearch],
});
```

The cached tool is a drop-in replacement -- it has the same name, description, parameters and approval settings as the original, plus a `.cache` object for management. Only its `execute` changes, so caching works the same inside agent runs and when you call the tool directly.

Whatever `execute` returns is cached, including error objects such as `{ error: 'API key not found…' }` that many built-in tools return instead of throwing. A thrown error is never cached. If a tool reports transient failures as data, wrap it so those results throw, or keep the TTL short.

***

## `withCache()` API

```typescript
function withCache<TParams, TResult>(
  tool: Tool<TParams, TResult>,
  config: WithCacheOptions
): CachedTool<TParams, TResult>;
```

### Configuration

```typescript
interface WithCacheOptions {
  strategy: 'exact' | 'semantic';
  ttl: DurationString;
  maxSize: number;
  storage: 'memory' | 'redis';
  similarity?: number;
  keyPrefix?: string;
  embeddingService?: EmbeddingService;
  redisClient?: RedisClientLike;
  onHit?: (key: string, params: unknown) => void;
  onMiss?: (key: string, params: unknown) => void;
  onEvict?: (key: string) => void;
}
```

| Field              | Type                    | Default       | Description                                       |
| ------------------ | ----------------------- | ------------- | ------------------------------------------------- |
| `strategy`         | `'exact' \| 'semantic'` | required      | Cache matching strategy                           |
| `ttl`              | `DurationString`        | required      | Time-to-live: `"30s"`, `"5m"`, `"2h"`, `"1d"`     |
| `maxSize`          | `number`                | required      | Maximum number of cached entries                  |
| `storage`          | `'memory' \| 'redis'`   | required      | Storage backend                                   |
| `similarity`       | `number`                | `0.95`        | Cosine similarity threshold for semantic matching |
| `keyPrefix`        | `string`                | `"toolcache"` | Key prefix; a trailing `:` is not doubled         |
| `embeddingService` | `EmbeddingService`      | --            | Required for semantic strategy                    |
| `redisClient`      | `RedisClientLike`       | --            | Required for redis storage (throws without it)    |
| `onHit`            | `function`              | --            | Called on an exact or semantic hit                |
| `onMiss`           | `function`              | --            | Called before the tool runs on a miss             |
| `onEvict`          | `function`              | --            | Called when an entry is invalidated or evicted    |

`maxSize` caps the number of entries per storage; when it is reached, the least recently used entry is dropped. Expired entries are dropped when they are read.

### Duration Strings

TTL values use human-readable duration strings:

| Suffix | Meaning      | Example   |
| ------ | ------------ | --------- |
| `ms`   | Milliseconds | `"500ms"` |
| `s`    | Seconds      | `"30s"`   |
| `m`    | Minutes      | `"10m"`   |
| `h`    | Hours        | `"2h"`    |
| `d`    | Days         | `"1d"`    |
| `w`    | Weeks        | `"1w"`    |

***

## Cache Strategies

### Exact Matching

Generates a deterministic cache key from the key prefix, the tool name and a SHA-256 hash of the parameters (serialized with sorted keys, so property order does not matter). Two calls with identical parameters hit the same cache entry.

```typescript
const cachedHash = withCache(hash, {
  strategy: 'exact',
  ttl: '1h',
  maxSize: 1000,
  storage: 'memory',
});
```

### Semantic Matching

Uses embedding vectors to find cached results for semantically similar inputs. Requires an `embeddingService` (any `EmbeddingService` from `@cogitator-ai/memory`, such as `OpenAIEmbeddingService`, or your own `{ embed, embedBatch, dimensions, model }`). Without one, `semantic` behaves like `exact`.

```typescript
const cachedSearch = withCache(webSearch, {
  strategy: 'semantic',
  ttl: '30m',
  maxSize: 200,
  storage: 'memory',
  similarity: 0.92,
  embeddingService: myEmbeddingService,
});
```

With semantic matching, a query like `"TypeScript best practices"` might match a cached result for `"best practices for TypeScript development"` if their cosine similarity exceeds the threshold.

The text that gets embedded is the first string among the parameters `query`, `q`, `search`, `text`, `input`, `prompt` and `question`, or the JSON of all parameters when there is none. A semantic hit therefore ignores the other parameters: `{ query: 'react hooks', maxResults: 20 }` can be answered from the entry for `{ query: 'React hooks', maxResults: 5 }`. Use `exact` when the other parameters change the result.

The lookup flow:

1. Check for an exact key match first
2. If no exact match, embed the current parameters and search for similar entries above the `similarity` threshold
3. If a similar entry is found, return its cached result
4. Otherwise, execute the tool and cache the result with its embedding

Embedding failures are ignored: the lookup falls back to running the tool, and the entry is stored without an embedding.

***

## Storage Backends

### In-Memory (LRU)

Entries are stored in a `Map` with LRU eviction -- when `maxSize` is reached, the least recently accessed entry is evicted. Each cached tool gets its own storage.

```typescript
const cached = withCache(sqlQuery, {
  strategy: 'exact',
  ttl: '5m',
  maxSize: 500,
  storage: 'memory',
});
```

In-memory storage is fast and requires no external dependencies, but entries are lost when the process restarts.

### Redis

For persistent, shared caching across multiple processes or server instances. Uses Redis sorted sets for LRU tracking and key-value storage for entries.

```typescript
import Redis from 'ioredis';
import { withCache, webScrape } from '@cogitator-ai/core';

const redis = new Redis('redis://localhost:6379');

const cached = withCache(webScrape, {
  strategy: 'exact',
  ttl: '1h',
  maxSize: 5000,
  storage: 'redis',
  redisClient: redis,
  keyPrefix: 'myapp:tools',
});
```

An ioredis client matches `RedisClientLike` as it is, and so does a client from [`@cogitator-ai/redis`](/docs/deployment/redis#tool-cache-client):

```typescript
import { createRedisClient } from '@cogitator-ai/redis';

const client = await createRedisClient({ url: 'redis://localhost:6379', keyPrefix: 'myapp:' });

const cachedScrape = withCache(webScrape, {
  strategy: 'exact',
  ttl: '1h',
  maxSize: 5000,
  storage: 'redis',
  redisClient: client,
});
```

Its `scan` is relative to the client's `keyPrefix` and walks every master in cluster mode, so `cache.clear()` and semantic lookups work in both modes; in a cluster, give the client a `{hashtag}` prefix so the storage's multi-key commands (`mget`, `del`) stay in one slot. Other clients need a small adapter: node-redis, for example, spells the commands `setEx`/`zAdd` and takes `SCAN` options as an object. Entries are written with `SETEX`, so Redis expires them on its own as well.

The storage keeps its data under `keyPrefix` (a `:` is appended when it does not end with one): each entry at `<prefix>:entry:<cache key>`, the LRU order in the sorted set `<prefix>:lru` and the entry count in `<prefix>:counter`. The cache key itself is `<keyPrefix>:<toolName>:<hash>` (again with a single `:` after the prefix), so with `keyPrefix: 'myapp:tools'` (the ioredis example) an entry lives at `myapp:tools:entry:myapp:tools:web_scrape:<hash>`. Cached tools that share a `keyPrefix` share one LRU and one `maxSize`, and `cache.clear()` deletes every key under the prefix. A `keyPrefix` set on the Redis client itself comes in front of all of these.

The `RedisClientLike` interface requires the methods used by Redis-backed cache storage:

```typescript
interface RedisClientLike {
  get(key: string): Promise<string | null>;
  setex(key: string, seconds: number, value: string): Promise<string>;
  del(...keys: string[]): Promise<number>;
  mget(...keys: string[]): Promise<(string | null)[]>;
  zadd(key: string, score: number, member: string): Promise<number>;
  zrange(key: string, start: number, stop: string): Promise<string[]>;
  zrem(key: string, ...members: string[]): Promise<number>;
  incr(key: string): Promise<number>;
  decr(key: string): Promise<number>;
  exists(...keys: string[]): Promise<number>;
  scan(
    cursor: number | string,
    matchToken: 'MATCH',
    pattern: string,
    countToken: 'COUNT',
    count: number | string
  ): Promise<[cursor: string, keys: string[]]>;
}
```

You can also create storage instances directly:

```typescript
import { createToolCacheStorage } from '@cogitator-ai/core';

const memoryStorage = createToolCacheStorage('memory', { maxSize: 1000 });

const redisStorage = createToolCacheStorage('redis', {
  redisClient: redis,
  keyPrefix: 'cache:',
  maxSize: 5000,
  onEvict: (key) => console.log('evicted', key),
});
```

`createToolCacheStorage(type, { maxSize?, redisClient?, keyPrefix?, onEvict? })` returns a `ToolCacheStorage` (`InMemoryToolCacheStorage` or `RedisToolCacheStorage`, both exported; `maxSize` defaults to 1000, the Redis key prefix to `toolcache:`). `onEvict` is called with the key of each entry evicted to make room for a new one. `withCache` creates its own storage from its options, so use these when you build caching around something other than a tool.

***

## Cache Management

Every cached tool exposes a `.cache` object for runtime management:

### Stats

```typescript
const stats = cachedSearch.cache.stats();
// {
//   hits: 142,
//   misses: 38,
//   size: 38,
//   evictions: 12,
//   hitRate: 0.789,
// }
```

### Clear

```typescript
await cachedSearch.cache.clear();
```

`hits`, `misses` and `hitRate` count calls of this cached tool since it was created; `evictions` counts entries dropped to stay under `maxSize`.

### Invalidate

Remove a specific entry by its parameters (it resolves to whether an entry existed, and calls `onEvict`):

```typescript
const existed = await cachedSearch.cache.invalidate({
  query: 'TypeScript tutorials',
  maxResults: 5,
});
```

### Warmup

Pre-populate the cache with known parameter-result pairs:

```typescript
await cachedSearch.cache.warmup([
  {
    params: { query: 'React hooks guide', maxResults: 5 },
    result: { query: 'React hooks guide', provider: 'tavily', results: [] },
  },
  {
    params: { query: 'Node.js streams', maxResults: 5 },
    result: { query: 'Node.js streams', provider: 'tavily', results: [] },
  },
]);
```

***

## When to Cache

Cache tools that:

* **Call external APIs** (web search, scraping) -- save quota and reduce latency
* **Run expensive queries** (SQL, vector search) -- avoid duplicate database load
* **Perform deterministic computation** (hashing, math) -- same input always gives same output
* **Have rate limits** -- cache prevents hitting API rate limits on repeated queries

Avoid caching tools that:

* **Have side effects** (file\_write, send\_email, exec) -- caching would silently skip the action
* **Must return real-time data** (datetime, random\_number) -- cached values would be stale
* **Depend on external state** that changes frequently between calls

***

## Observability Callbacks

Use the `onHit`, `onMiss`, and `onEvict` callbacks to integrate with your monitoring. `onEvict` receives the cache key of each entry removed by `cache.invalidate()` or dropped to stay under `maxSize` (in memory and in Redis); expired entries are dropped silently.

```typescript
const cached = withCache(webSearch, {
  strategy: 'exact',
  ttl: '10m',
  maxSize: 500,
  storage: 'memory',
  onHit: (key, params) => {
    metrics.increment('tool_cache.hit', { tool: 'web_search' });
  },
  onMiss: (key, params) => {
    metrics.increment('tool_cache.miss', { tool: 'web_search' });
  },
  onEvict: (key) => {
    metrics.increment('tool_cache.evict', { tool: 'web_search' });
  },
});
```

## Helpers

| Export                                            | Description                                                    |
| ------------------------------------------------- | -------------------------------------------------------------- |
| `generateCacheKey({ toolName, params, prefix? })` | The exact-match key: `<prefix>:<toolName>:<16-hex-char hash>`  |
| `paramsToQueryString(params)`                     | The text semantic matching embeds                              |
| `cosineSimilarity(a, b)`                          | Similarity of two vectors                                      |
| `parseDuration(duration)`                         | `DurationString` to milliseconds (throws on an invalid string) |