LLM Backends

OverviewCopy Anchor Link

Cogitator supports 11 LLM providers out of the box. Backends are created lazily -- a provider's client is only instantiated the first time an agent needs it. The provider is determined by the model string prefix.

const cog = new Cogitator({
  llm: {
    defaultProvider: 'openai',
    providers: {
      openai: { apiKey: process.env.OPENAI_API_KEY! },
      anthropic: { apiKey: process.env.ANTHROPIC_API_KEY! },
      ollama: { baseUrl: 'http://localhost:11434' },
    },
  },
});

Provider RoutingCopy Anchor Link

Model strings follow the provider/model-name format:

'openai/gpt-6.1-sol'; // routes to OpenAI
'anthropic/claude-sonnet-5-5'; // routes to Anthropic
'ollama/llama3.3:latest'; // routes to Ollama
'google/gemini-3.8-flash'; // routes to Google

If the model string has no prefix, or its prefix is not a built-in provider, a backend in llm.backends or a registered plugin, the whole string goes to defaultProvider (or ollama if unset). That is how meta-llama/llama-3.3-70b reaches an OpenAI-compatible default provider unchanged. An agent's provider field skips the prefix check and sends the model string as it is.

The parseModel() utility handles this splitting:

import { parseModel } from '@cogitator-ai/core';

parseModel('openai/gpt-6.1-sol');
// { provider: 'openai', model: 'gpt-6.1-sol' }

parseModel('llama3.3:latest');
// { provider: null, model: 'llama3.3:latest' }

Supported ProvidersCopy Anchor Link

OllamaCopy Anchor Link

Local inference with Ollama. No API key required.

providers: {
  ollama: {
    baseUrl: 'http://localhost:11434', // default
    apiKey: process.env.OLLAMA_API_KEY, // optional: Ollama Cloud (https://ollama.com) or an authenticated server
  },
}

model: 'ollama/llama3.3:latest';
model: 'ollama/codellama:34b';
model: 'ollama/mistral:7b-instruct';

OpenAICopy Anchor Link

providers: {
  openai: {
    apiKey: process.env.OPENAI_API_KEY!,
    baseUrl: 'https://api.openai.com/v1', // optional, for proxies
    api: 'responses', // optional: 'responses' | 'chat-completions'
  },
}

model: 'openai/gpt-6.1-sol'; // default when no model is given
model: 'openai/gpt-6-luna';
model: 'openai/gpt-6-astra';

The official OpenAI backend talks to the Responses API. GPT-6 models only support tool calling there, and GPT-5.x/GPT-6 reject non-default sampling parameters on Chat Completions. Requests are stateless (store: false); encrypted reasoning items are kept on ToolCall.replay and sent back with the call on the next turn, so reasoning models keep their chain of thought across tool calls (also after a memory round-trip).

OpenAI-compatible providers (Azure, Mistral, Groq, Together, DeepSeek, vLLM, any custom baseUrl) stay on Chat Completions unchanged. DEFAULT_OPENAI_MODEL and isOpenAIReasoningModel(model) are exported from @cogitator-ai/core.

AnthropicCopy Anchor Link

providers: {
  anthropic: {
    apiKey: process.env.ANTHROPIC_API_KEY!,
  },
}

model: 'anthropic/claude-sonnet-5-5'; // default when no model is given
model: 'anthropic/claude-opus-5-5';
model: 'anthropic/claude-haiku-4-5';

The backend adapts each request to what the target Claude model accepts (model ids from the Anthropic API, Vertex AI and Bedrock, including inference-profile prefixes and ARNs, are recognised):

The stop reasons model_context_window_exceeded and pause_turn are reported as finishReason: 'length', and refusal as 'error'.

Google (Gemini)Copy Anchor Link

providers: {
  google: {
    apiKey: process.env.GOOGLE_API_KEY!,
  },
}

model: 'google/gemini-3.8-flash';
model: 'google/gemini-3.1-pro-preview';
model: 'google/gemini-3.5-flash-lite';

Gemini has no null schema type, so .nullable() fields in tool parameters and response schemas are sent as nullable: true. Gemini attaches a thought signature to function calls; it is stored on ToolCall.thoughtSignature (also in memory) and sent back with the call on the next request, which Gemini 3 models require for multi-step tool use. Thought-summary parts are excluded from the response text.

Azure OpenAICopy Anchor Link

providers: {
  azure: {
    endpoint: 'https://your-resource.openai.azure.com',
    apiKey: process.env.AZURE_OPENAI_API_KEY!,
    apiVersion: '2024-02-01',  // optional
    deployment: 'gpt-6.1-sol',      // optional default deployment
  },
}

AWS BedrockCopy Anchor Link

providers: {
  bedrock: {
    region: 'us-east-1',
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
}

Bedrock needs the optional @aws-sdk/client-bedrock-runtime package. Without explicit credentials it uses the AWS credential chain.

model: 'bedrock/global.anthropic.claude-sonnet-5-5';
model: 'bedrock/global.anthropic.claude-haiku-4-5-20251001-v1:0';

Claude models on Bedrock follow the same sampling-parameter and forced-tool-choice rules as the Anthropic backend. json_object and json_schema response formats are supported. json_object is requested through a system-prompt instruction; json_schema is enforced with Converse outputConfig.textFormat on Claude 4.5 – 4.6 (the only Claude versions Bedrock serves structured outputs for) and otherwise sent as a system-prompt instruction that includes the schema.

OpenAI-Compatible ProvidersCopy Anchor Link

Mistral, Groq, Together, DeepSeek, and vLLM use the OpenAI SDK under the hood with a provider-specific base URL. Hosted providers only need an API key; vLLM needs the base URL of your OpenAI-compatible server.

providers: {
  mistral: { apiKey: process.env.MISTRAL_API_KEY! },
  groq: { apiKey: process.env.GROQ_API_KEY! },
  together: { apiKey: process.env.TOGETHER_API_KEY! },
  deepseek: { apiKey: process.env.DEEPSEEK_API_KEY! },
  vllm: { baseUrl: 'http://localhost:8000/v1' },
}

model: 'mistral/mistral-large-latest';
model: 'groq/llama-3.3-70b-versatile';
model: 'together/meta-llama/Llama-3.3-70B-Instruct-Turbo';
model: 'deepseek/deepseek-chat';
model: 'vllm/mistral-7b';

Message ConversionCopy Anchor Link

All backends receive the same Message[]. Every system message (the agent instructions plus any later ones, such as reflection hints or context summaries) is combined into the provider's system prompt for Anthropic, Gemini and Bedrock, and consecutive tool results are sent together in a single turn, as those APIs expect for parallel tool calls. Images in tool results are placed where each API accepts them (see Images in Tool Results).

Custom BackendsCopy Anchor Link

Backend InstancesCopy Anchor Link

A backend implements LLMBackend (provider, chat(), chatStream()), usually by extending BaseLLMBackend, or OpenAICompatibleBackend for servers that speak the OpenAI API. Its provider field is an LLMBackendProvider: a built-in provider name or any name of your own (usually the one it is registered under). Routing goes by the name in llm.backends or the plugin's provider, not by this field.

Pass backends of your own to llm.backends. An agent whose model starts with the name, or whose provider is the name, runs on it; the rest of the model string is sent as the model name. A name of a built-in provider replaces it, for example with a withDebug() wrapper:

const cog = new Cogitator({
  llm: {
    backends: {
      local: new MyCustomBackend({ url: 'http://gpu-box:8000' }),
      openai: withDebug(new OpenAIBackend({ apiKey: process.env.OPENAI_API_KEY! })),
    },
  },
});

const agent = new Agent({ name: 'a', model: 'local/my-finetune', instructions: '...' });

cog.route('local/my-finetune') returns the backend and model name a run would use. A run that names a provider nobody provides fails with CONFIGURATION_ERROR. cog.knowsProvider('local') tells whether a local/... prefix routes to that provider (a backend in llm.backends, a built-in provider or a registered plugin) or stays part of the model name on llm.defaultProvider.

Plugin SystemCopy Anchor Link

Register custom LLM backends for providers not included by default. The runtime creates a plugin's backend the first time an agent uses its provider name, with the config under that name in llm.plugins:

import { defineBackend, registerLLMBackend } from '@cogitator-ai/core';

const myBackend = defineBackend<{ apiKey: string }>({
  provider: 'my-provider',
  metadata: {
    name: 'My Provider', // version, description and author are optional
    version: '1.0.0',
  },
  create: (config) => new MyCustomBackend(config),
  validateConfig: (config): config is { apiKey: string } =>
    typeof config === 'object' && config !== null && 'apiKey' in config,
});

registerLLMBackend(myBackend);

const cog = new Cogitator({ llm: { plugins: { 'my-provider': { apiKey: '...' } } } });
const agent = new Agent({ name: 'a', model: 'my-provider/model-x', instructions: '...' });

Registering a provider name twice throws; unregisterLLMBackend(provider) removes one. A failing validateConfig makes the first run on that provider fail with Invalid configuration for provider '...'.

List registered plugins:

import { listLLMPlugins, hasLLMPlugin } from '@cogitator-ai/core';

console.log(listLLMPlugins());
console.log(hasLLMPlugin('my-provider'));

Debug WrapperCopy Anchor Link

Wrap any backend with debug logging to inspect requests and responses:

import { withDebug } from '@cogitator-ai/core';

const debugBackend = withDebug(backend, {
  logRequest: true,
  logResponse: true,
  logStream: false,
  redactApiKeys: true,
  maxContentLength: 2000,
  logger: {
    log: (level, message, data) => console.log(`[LLM:${level}]`, message, data ?? ''),
  },
});

Without a logger it prints timestamped entries to the console. Put the wrapped backend into llm.backends under the provider's name to debug every run on it.

ReasoningCopy Anchor Link

reasoning on an agent (or a run, or a ChatRequest) controls reasoning models the same way on every provider:

reasoning: {
  effort: 'high',      // none | minimal | low | medium | high | xhigh | max
  budgetTokens: 8000,  // providers that take a thinking budget
  summary: true,       // return a readable summary of the reasoning
}

The reasoning comes back as ChatResponse.reasoning and delta.reasoning while streaming; agent runs collect it into RunResult.reasoning and stream it to onReasoning. Reasoning tokens are reported in usage.reasoningTokens and counted in outputTokens (Gemini bills them as output too). Summaries are best-effort on some providers: Gemini omits the thought summary from a fair share of streamed responses even when it thought (reasoningTokens is still set), while non-streamed responses carry it.

Reasoning across tool calls. Claude and OpenAI reasoning models keep thinking between tool calls only if their reasoning comes back with the next request. Backends keep it with the tool calls it preceded (ToolCall.replay) and send it back while the agent works through a tool loop. Claude's thinking blocks are tied to the conversation that produced them, so they are replayed only after the latest user message; if history was rewritten in between (context compression), the backend retries once without them.

Prompt CachingCopy Anchor Link

Agent runs cache their prompt by default. Anthropic and Bedrock cache only what the request marks, so their backends mark the stable prefix (tools, system prompt, history); OpenAI and Gemini cache on their own. Every provider reports what it served from cache, and the run's cost prices those tokens at the model's cache price:

const result = await cog.run(agent, { input });
result.usage.cachedInputTokens; // read from the cache (about a tenth of the input price on Claude)
result.usage.cacheWriteTokens; // written to it (Anthropic: 1.25× the input price)

In an agent loop every iteration resends the previous one's prompt, so after the first call most of the input is a cache read. llm.promptCache: { ttl: '1h' } keeps Anthropic's cache for an hour instead of five minutes; llm.promptCache: false turns marking off. A single ChatRequest takes the same setting as cache.

RetriesCopy Anchor Link

The runtime retries LLM calls that fail with a retryable error: rate limits (429), server errors (5xx), timeouts and dropped connections. Authentication, bad requests and content filters fail at once. Retries are on by default for every backend an agent runs on, including your own llm.backends and plugins:

const cog = new Cogitator({
  llm: {
    defaultModel: 'anthropic/claude-sonnet-5-5',
    retry: {
      maxRetries: 3, // attempts after the first one (default 2)
      baseDelay: 1000, // first backoff, doubled on each retry (default 1000 ms)
      maxDelay: 30_000, // backoff cap (default 30000 ms)
      maxRetryAfter: 60_000, // longest Retry-After worth waiting for (default 60000 ms)
      onRetry: ({ provider, attempt, delay, error }) =>
        console.warn(`${provider}: retry ${attempt} in ${delay}ms (${error.message})`),
    },
  },
});

• Retry-After. When the provider names a wait (retry-after, retry-after-ms or Gemini's RetryInfo), the runtime waits exactly that long. If the wait is longer than maxRetryAfter, the call fails at once instead of holding the run.
• Streaming. A stream is retried only until its first chunk. After that the caller has already seen part of the answer, so the error is passed on.
• Cancellation. The run's timeout and abort signal stop the waits between attempts too.
• One layer. Backends created by the runtime make one attempt per call: the OpenAI, Anthropic, Azure and Bedrock SDKs have their own retries switched off, so attempts never multiply.

retry: false turns retries off. To use a backend on its own with the same behaviour, wrap it with withLLMRetry(backend, { maxRetries: 3 }).

Error HandlingCopy Anchor Link

All backends throw typed LLMError instances with structured error codes:

import { LLMError, ErrorCode } from '@cogitator-ai/core';

try {
  await cog.run(agent, { input: 'Hello' });
} catch (e) {
  if (e instanceof LLMError) {
    if (e.code === ErrorCode.LLM_RATE_LIMITED) {
      // still rate limited after the runtime's retries; e.retryAfter is the provider's hint, if any
    }
  }
}

Error codes include LLM_UNAVAILABLE, LLM_RATE_LIMITED, LLM_TIMEOUT, LLM_INVALID_RESPONSE, LLM_CONTEXT_LENGTH_EXCEEDED and LLM_CONTENT_FILTERED; e.details carries the provider, model, status code and request id. Rate limit, server and connection errors are marked as retryable, and retryAfter holds the wait the provider asked for, when it named one.