# LLM Backends URL: https://cogitator.app/docs/core/llm-backends > Supported LLM providers, configuration, and provider-prefix routing. ## Overview Cogitator supports 11 LLM providers out of the box. Backends are created lazily -- a provider's client is only instantiated the first time an agent needs it. The provider is determined by the model string prefix. ```typescript const cog = new Cogitator({ llm: { defaultProvider: 'openai', providers: { openai: { apiKey: process.env.OPENAI_API_KEY! }, anthropic: { apiKey: process.env.ANTHROPIC_API_KEY! }, ollama: { baseUrl: 'http://localhost:11434' }, }, }, }); ``` ## Provider Routing Model strings follow the `provider/model-name` format: ```typescript 'openai/gpt-6.1-sol'; // routes to OpenAI 'anthropic/claude-sonnet-5-5'; // routes to Anthropic 'ollama/llama3.3:latest'; // routes to Ollama 'google/gemini-3.8-flash'; // routes to Google ``` If the model string has no prefix, or its prefix is not a built-in provider, a backend in `llm.backends` or a registered plugin, the whole string goes to `defaultProvider` (or `ollama` if unset). That is how `meta-llama/llama-3.3-70b` reaches an OpenAI-compatible default provider unchanged. An agent's `provider` field skips the prefix check and sends the model string as it is. The `parseModel()` utility handles this splitting: ```typescript import { parseModel } from '@cogitator-ai/core'; parseModel('openai/gpt-6.1-sol'); // { provider: 'openai', model: 'gpt-6.1-sol' } parseModel('llama3.3:latest'); // { provider: null, model: 'llama3.3:latest' } ``` ## Supported Providers ### Ollama Local inference with Ollama. No API key required. ```typescript providers: { ollama: { baseUrl: 'http://localhost:11434', // default apiKey: process.env.OLLAMA_API_KEY, // optional: Ollama Cloud (https://ollama.com) or an authenticated server }, } ``` ```typescript model: 'ollama/llama3.3:latest'; model: 'ollama/codellama:34b'; model: 'ollama/mistral:7b-instruct'; ``` ### OpenAI ```typescript providers: { openai: { apiKey: process.env.OPENAI_API_KEY!, baseUrl: 'https://api.openai.com/v1', // optional, for proxies api: 'responses', // optional: 'responses' | 'chat-completions' }, } ``` ```typescript model: 'openai/gpt-6.1-sol'; // default when no model is given model: 'openai/gpt-6-luna'; model: 'openai/gpt-6-astra'; ``` The official OpenAI backend talks to the **Responses API**. GPT-6 models only support tool calling there, and GPT-5.x/GPT-6 reject non-default sampling parameters on Chat Completions. Requests are stateless (`store: false`); encrypted reasoning items are kept on `ToolCall.replay` and sent back with the call on the next turn, so reasoning models keep their chain of thought across tool calls (also after a memory round-trip). | Behaviour | Details | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Wire API selection | `responses` for the official endpoint (no `baseUrl`, or a `baseUrl` on `api.openai.com`); `chat-completions` when `baseUrl` points anywhere else. `api` forces either path | | Stop sequences | The Responses API has no stop parameter, so a request with `stop` sequences is sent through Chat Completions | | Sampling params | Reasoning models (o-series, GPT-5 and later) get no `temperature` / `top_p`; `gpt-4o` / `gpt-4.1` and `-chat` variants keep them | | `maxTokens` | Sent as `max_output_tokens` on the Responses API; on Chat Completions as `max_completion_tokens` for the official endpoint and `max_tokens` otherwise | | Structured output | `responseFormat` maps to `text.format` (`json_object` / `json_schema`); images are sent as `input_image` | | Usage | `usage.cachedInputTokens` and `usage.reasoningTokens` are reported when OpenAI returns them (both already included in `inputTokens` / `outputTokens`) | OpenAI-compatible providers (Azure, Mistral, Groq, Together, DeepSeek, vLLM, any custom `baseUrl`) stay on Chat Completions unchanged. `DEFAULT_OPENAI_MODEL` and `isOpenAIReasoningModel(model)` are exported from `@cogitator-ai/core`. ### Anthropic ```typescript providers: { anthropic: { apiKey: process.env.ANTHROPIC_API_KEY!, }, } ``` ```typescript model: 'anthropic/claude-sonnet-5-5'; // default when no model is given model: 'anthropic/claude-opus-5-5'; model: 'anthropic/claude-haiku-4-5'; ``` The backend adapts each request to what the target Claude model accepts (model ids from the Anthropic API, Vertex AI and Bedrock, including inference-profile prefixes and ARNs, are recognised): | Model family | Behaviour | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Claude 4.7+, 5.x, Fable | `temperature` / `top_p` are omitted (these models reject non-default sampling params), including the agent default `0.7` | | Claude 4.0 – 4.6 | At most one of `temperature` / `top_p` is sent; `temperature` wins when both are set | | Claude 4.5+, Fable | `json_schema` response format uses native structured outputs (`output_config`) instead of a forced tool | | Claude Opus 5.5, Sonnet 5.5, Fable | Forced tool choice (`'required'` or a named function) falls back to `auto` with a system-prompt instruction naming the tool, plus a one-time warning per model | The stop reasons `model_context_window_exceeded` and `pause_turn` are reported as `finishReason: 'length'`, and `refusal` as `'error'`. ### Google (Gemini) ```typescript providers: { google: { apiKey: process.env.GOOGLE_API_KEY!, }, } ``` ```typescript model: 'google/gemini-3.8-flash'; model: 'google/gemini-3.1-pro-preview'; model: 'google/gemini-3.5-flash-lite'; ``` Gemini has no `null` schema type, so `.nullable()` fields in tool parameters and response schemas are sent as `nullable: true`. Gemini attaches a thought signature to function calls; it is stored on `ToolCall.thoughtSignature` (also in memory) and sent back with the call on the next request, which Gemini 3 models require for multi-step tool use. Thought-summary parts are excluded from the response text. ### Azure OpenAI ```typescript providers: { azure: { endpoint: 'https://your-resource.openai.azure.com', apiKey: process.env.AZURE_OPENAI_API_KEY!, apiVersion: '2024-02-01', // optional deployment: 'gpt-6.1-sol', // optional default deployment }, } ``` ### AWS Bedrock ```typescript providers: { bedrock: { region: 'us-east-1', accessKeyId: process.env.AWS_ACCESS_KEY_ID, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, }, } ``` Bedrock needs the optional `@aws-sdk/client-bedrock-runtime` package. Without explicit credentials it uses the AWS credential chain. ```typescript model: 'bedrock/global.anthropic.claude-sonnet-5-5'; model: 'bedrock/global.anthropic.claude-haiku-4-5-20251001-v1:0'; ``` Claude models on Bedrock follow the same sampling-parameter and forced-tool-choice rules as the Anthropic backend. `json_object` and `json_schema` response formats are supported. `json_object` is requested through a system-prompt instruction; `json_schema` is enforced with Converse `outputConfig.textFormat` on Claude 4.5 – 4.6 (the only Claude versions Bedrock serves structured outputs for) and otherwise sent as a system-prompt instruction that includes the schema. ### OpenAI-Compatible Providers Mistral, Groq, Together, DeepSeek, and vLLM use the OpenAI SDK under the hood with a provider-specific base URL. Hosted providers only need an API key; vLLM needs the base URL of your OpenAI-compatible server. ```typescript providers: { mistral: { apiKey: process.env.MISTRAL_API_KEY! }, groq: { apiKey: process.env.GROQ_API_KEY! }, together: { apiKey: process.env.TOGETHER_API_KEY! }, deepseek: { apiKey: process.env.DEEPSEEK_API_KEY! }, vllm: { baseUrl: 'http://localhost:8000/v1' }, } ``` ```typescript model: 'mistral/mistral-large-latest'; model: 'groq/llama-3.3-70b-versatile'; model: 'together/meta-llama/Llama-3.3-70B-Instruct-Turbo'; model: 'deepseek/deepseek-chat'; model: 'vllm/mistral-7b'; ``` ### Message Conversion All backends receive the same `Message[]`. Every `system` message (the agent instructions plus any later ones, such as reflection hints or context summaries) is combined into the provider's system prompt for Anthropic, Gemini and Bedrock, and consecutive tool results are sent together in a single turn, as those APIs expect for parallel tool calls. Images in tool results are placed where each API accepts them (see [Images in Tool Results](/docs/core/tools#images-in-tool-results)). ## Custom Backends ### Backend Instances A backend implements `LLMBackend` (`provider`, `chat()`, `chatStream()`), usually by extending `BaseLLMBackend`, or `OpenAICompatibleBackend` for servers that speak the OpenAI API. Its `provider` field is an `LLMBackendProvider`: a built-in provider name or any name of your own (usually the one it is registered under). Routing goes by the name in `llm.backends` or the plugin's provider, not by this field. Pass backends of your own to `llm.backends`. An agent whose model starts with the name, or whose `provider` is the name, runs on it; the rest of the model string is sent as the model name. A name of a built-in provider replaces it, for example with a `withDebug()` wrapper: ```typescript const cog = new Cogitator({ llm: { backends: { local: new MyCustomBackend({ url: 'http://gpu-box:8000' }), openai: withDebug(new OpenAIBackend({ apiKey: process.env.OPENAI_API_KEY! })), }, }, }); const agent = new Agent({ name: 'a', model: 'local/my-finetune', instructions: '...' }); ``` `cog.route('local/my-finetune')` returns the backend and model name a run would use. A run that names a provider nobody provides fails with `CONFIGURATION_ERROR`. `cog.knowsProvider('local')` tells whether a `local/...` prefix routes to that provider (a backend in `llm.backends`, a built-in provider or a registered plugin) or stays part of the model name on `llm.defaultProvider`. ### Plugin System Register custom LLM backends for providers not included by default. The runtime creates a plugin's backend the first time an agent uses its provider name, with the config under that name in `llm.plugins`: ```typescript import { defineBackend, registerLLMBackend } from '@cogitator-ai/core'; const myBackend = defineBackend<{ apiKey: string }>({ provider: 'my-provider', metadata: { name: 'My Provider', // version, description and author are optional version: '1.0.0', }, create: (config) => new MyCustomBackend(config), validateConfig: (config): config is { apiKey: string } => typeof config === 'object' && config !== null && 'apiKey' in config, }); registerLLMBackend(myBackend); const cog = new Cogitator({ llm: { plugins: { 'my-provider': { apiKey: '...' } } } }); const agent = new Agent({ name: 'a', model: 'my-provider/model-x', instructions: '...' }); ``` Registering a provider name twice throws; `unregisterLLMBackend(provider)` removes one. A failing `validateConfig` makes the first run on that provider fail with `Invalid configuration for provider '...'`. List registered plugins: ```typescript import { listLLMPlugins, hasLLMPlugin } from '@cogitator-ai/core'; console.log(listLLMPlugins()); console.log(hasLLMPlugin('my-provider')); ``` ## Debug Wrapper Wrap any backend with debug logging to inspect requests and responses: ```typescript import { withDebug } from '@cogitator-ai/core'; const debugBackend = withDebug(backend, { logRequest: true, logResponse: true, logStream: false, redactApiKeys: true, maxContentLength: 2000, logger: { log: (level, message, data) => console.log(`[LLM:${level}]`, message, data ?? ''), }, }); ``` Without a `logger` it prints timestamped entries to the console. Put the wrapped backend into `llm.backends` under the provider's name to debug every run on it. ## Reasoning `reasoning` on an agent (or a run, or a `ChatRequest`) controls reasoning models the same way on every provider: ```typescript reasoning: { effort: 'high', // none | minimal | low | medium | high | xhigh | max budgetTokens: 8000, // providers that take a thinking budget summary: true, // return a readable summary of the reasoning } ``` | Provider | What `reasoning` becomes | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Anthropic, Bedrock | Claude 4.6+ / 5.x / Fable: `thinking: { type: 'adaptive' }` and the closest `output_config.effort` the model accepts; `none` turns thinking off where the model allows it (`disabled`, or `between_tools` on Sonnet 5.5) and asks for the lowest effort where it does not (Opus 5.5, Fable). Claude 3.7 – 4.5: a thinking budget (`budgetTokens`, or derived from the effort) added on top of `maxTokens`. | | OpenAI (Responses) | `reasoning.effort` and, with `summary`, `reasoning.summary: 'auto'` — reasoning models only | | OpenAI-compatible | `reasoning_effort`; reasoning text comes from `reasoning_content` / `reasoning` (DeepSeek, vLLM, Groq, OpenRouter) | | Google | Gemini 3+: `thinkingConfig.thinkingLevel`; Gemini 2.5: `thinkingBudget` (Pro never below 128); `summary` → `includeThoughts` | | Ollama | `think`: a level for gpt-oss, on or off for other thinking models | The reasoning comes back as `ChatResponse.reasoning` and `delta.reasoning` while streaming; agent runs collect it into `RunResult.reasoning` and stream it to `onReasoning`. Reasoning tokens are reported in `usage.reasoningTokens` and counted in `outputTokens` (Gemini bills them as output too). Summaries are best-effort on some providers: Gemini omits the thought summary from a fair share of streamed responses even when it thought (`reasoningTokens` is still set), while non-streamed responses carry it. **Reasoning across tool calls.** Claude and OpenAI reasoning models keep thinking between tool calls only if their reasoning comes back with the next request. Backends keep it with the tool calls it preceded (`ToolCall.replay`) and send it back while the agent works through a tool loop. Claude's thinking blocks are tied to the conversation that produced them, so they are replayed only after the latest user message; if history was rewritten in between (context compression), the backend retries once without them. ## Prompt Caching Agent runs cache their prompt by default. Anthropic and Bedrock cache only what the request marks, so their backends mark the stable prefix (tools, system prompt, history); OpenAI and Gemini cache on their own. Every provider reports what it served from cache, and the run's cost prices those tokens at the model's cache price: ```typescript const result = await cog.run(agent, { input }); result.usage.cachedInputTokens; // read from the cache (about a tenth of the input price on Claude) result.usage.cacheWriteTokens; // written to it (Anthropic: 1.25× the input price) ``` In an agent loop every iteration resends the previous one's prompt, so after the first call most of the input is a cache read. `llm.promptCache: { ttl: '1h' }` keeps Anthropic's cache for an hour instead of five minutes; `llm.promptCache: false` turns marking off. A single `ChatRequest` takes the same setting as `cache`. ## Retries The runtime retries LLM calls that fail with a retryable error: rate limits (429), server errors (5xx), timeouts and dropped connections. Authentication, bad requests and content filters fail at once. Retries are on by default for every backend an agent runs on, including your own `llm.backends` and plugins: ```typescript const cog = new Cogitator({ llm: { defaultModel: 'anthropic/claude-sonnet-5-5', retry: { maxRetries: 3, // attempts after the first one (default 2) baseDelay: 1000, // first backoff, doubled on each retry (default 1000 ms) maxDelay: 30_000, // backoff cap (default 30000 ms) maxRetryAfter: 60_000, // longest Retry-After worth waiting for (default 60000 ms) onRetry: ({ provider, attempt, delay, error }) => console.warn(`${provider}: retry ${attempt} in ${delay}ms (${error.message})`), }, }, }); ``` * **Retry-After.** When the provider names a wait (`retry-after`, `retry-after-ms` or Gemini's `RetryInfo`), the runtime waits exactly that long. If the wait is longer than `maxRetryAfter`, the call fails at once instead of holding the run. * **Streaming.** A stream is retried only until its first chunk. After that the caller has already seen part of the answer, so the error is passed on. * **Cancellation.** The run's timeout and abort signal stop the waits between attempts too. * **One layer.** Backends created by the runtime make one attempt per call: the OpenAI, Anthropic, Azure and Bedrock SDKs have their own retries switched off, so attempts never multiply. `retry: false` turns retries off. To use a backend on its own with the same behaviour, wrap it with `withLLMRetry(backend, { maxRetries: 3 })`. ## Error Handling All backends throw typed `LLMError` instances with structured error codes: ```typescript import { LLMError, ErrorCode } from '@cogitator-ai/core'; try { await cog.run(agent, { input: 'Hello' }); } catch (e) { if (e instanceof LLMError) { if (e.code === ErrorCode.LLM_RATE_LIMITED) { // still rate limited after the runtime's retries; e.retryAfter is the provider's hint, if any } } } ``` Error codes include `LLM_UNAVAILABLE`, `LLM_RATE_LIMITED`, `LLM_TIMEOUT`, `LLM_INVALID_RESPONSE`, `LLM_CONTEXT_LENGTH_EXCEEDED` and `LLM_CONTENT_FILTERED`; `e.details` carries the provider, model, status code and request id. Rate limit, server and connection errors are marked as retryable, and `retryAfter` holds the wait the provider asked for, when it named one.