# Distributed Swarms URL: https://cogitator.app/docs/swarms/distributed > Scale swarms across multiple machines with Redis-backed coordination, distributed worker nodes, and shared state. ## Overview By default, swarms run in-memory on a single process. For production workloads, you can enable **distributed mode** to coordinate agents across multiple machines using Redis as the communication backbone. ``` ┌─────────────────────────┐ │ Swarm Coordinator │ │ (orchestrator node) │ │ │ │ Strategy ──▶ Redis │ │ (pub/sub) │ └──────────┬──────────────┘ │ job queue ┌──────┴──────┐ ▼ ▼ ┌─────────┐ ┌─────────┐ │ Worker │ │ Worker │ │ Node 1 │ │ Node 2 │ │ │ │ │ │ Cogitator│ │ Cogitator│ │ + Agent │ │ + Agent │ └─────────┘ └─────────┘ ``` The orchestrator node runs the strategy and dispatches agent jobs to a Redis queue. Worker nodes pick up jobs, execute the agent, and publish results back through Redis. ## Enabling Distributed Mode ```typescript import { Cogitator, Agent } from '@cogitator-ai/core'; import { swarm } from '@cogitator-ai/swarms'; const model = 'ollama/llama3.2'; const team = swarm('distributed-team') .strategy('pipeline') .pipeline({ stages: [ { name: 'draft', agent: new Agent({ name: 'writer', model, instructions: 'Draft the text.' }), }, { name: 'review', agent: new Agent({ name: 'editor', model, instructions: 'Review it.' }) }, ], }) .distributed({ enabled: true, redis: { host: 'redis.example.com', port: 6379, password: process.env.REDIS_PASSWORD, db: 0, keyPrefix: 'swarm', }, queue: 'swarm-agent-jobs', timeout: 300_000, }) .build(cogitator); await team.run({ input: 'Write the release notes' }); await team.close(); ``` Always call `team.close()` when done: it closes the Redis connections and schedules the swarm's Redis state to expire after `cleanupAfter`. ## Distributed Configuration | Option | Default | Description | | ----------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | `false` | Enable distributed mode | | `redis.host` | `'localhost'` | Redis server hostname | | `redis.port` | `6379` | Redis server port | | `redis.password` | `undefined` | Redis authentication password | | `redis.db` | `0` | Redis database number | | `redis.keyPrefix` | `'swarm'` | Key prefix for all Redis keys | | `queue` | `'swarm-agent-jobs'` | Job queue name (`{keyPrefix}:jobs:{queue}`) | | `timeout` | `300000` | Timeout in ms for waiting on agent job results | | `retry` | `undefined` (no retries) | Re-dispatch jobs that fail on a worker or time out: `maxRetries` (`3`), `backoff` (`'exponential'`; or `'constant'`, `'linear'`), `initialDelay` (`1000` ms), `maxDelay` (`30000` ms) | | `cleanupAfter` | `3600000` | How long the swarm's Redis state (blackboard, messages, events) is kept after `close()`, in ms; `0` deletes it at once | `retry` acts before the swarm's `errorHandling` sees the failure, and an aborted run is never retried. A timed-out job may still be running on its worker, so a retried turn can run twice. `workerConcurrency` is deprecated and has no effect: set concurrency where the workers run, with `DistributedSwarmWorker`'s `concurrency` option. ```typescript swarm('team') .strategy('pipeline') .pipeline({ stages }) .distributed({ enabled: true, redis: { host: 'localhost', port: 6379 }, retry: { maxRetries: 2, backoff: 'exponential', initialDelay: 500 }, cleanupAfter: 600_000, }); ``` ## How It Works When distributed mode is enabled, the `Swarm` creates a `DistributedSwarmCoordinator` instead of the local `SwarmCoordinator`. The distributed coordinator: 1. **Uses Redis-backed primitives** — `RedisMessageBus`, `RedisBlackboard`, and `RedisSwarmEventEmitter` replace their in-memory counterparts. They are available right after construction and connect lazily on the first run. 2. **Dispatches jobs** — when a strategy runs an agent, the coordinator serializes the agent (model, instructions, limits and tool schemas), the full turn context and run options, and pushes a `SwarmAgentJobPayload` onto the Redis list `{keyPrefix}:jobs:{queue}` (see `swarmJobQueueKey()`). 3. **Waits for results** — results arrive on the `{keyPrefix}:{swarmId}:results` pub/sub channel and are matched by job id. Turns time out after `timeout` ms and are cancelled when the swarm is aborted or the run times out. 4. **Applies the same policies as local swarms** — retry, failover, skip, budgets, circuit breaking and events behave exactly like in-process execution. 5. **Synchronizes state** — blackboard writes and messages are persisted in Redis, so all nodes see the same state. The worker runs each turn on the model the agent would use in-process. The payload carries the agent's model (or the swarm's `resolveModel` default), prefixed with the agent's own `provider` when it sets one, and the worker routes it with its own `cogitator`: a model like `openrouter/deepseek/deepseek-v4-pro` runs on the worker's `openrouter` backend. Give the worker the same `llm.backends`, plugins and provider keys as the process that runs the swarm. A provider it cannot route to fails the turn. Strategy tools that call back into the coordinator (hierarchical delegation, voting, negotiation tools) only exist in-process: a distributed hierarchical supervisor cannot delegate, distributed negotiators cannot make offers, and consensus voters vote with a `VOTE:` line. `agentTools` is rejected for distributed swarms. Prefer pipeline, round-robin, consensus, debate and auction for distributed swarms. Agent tools are executed on the worker from its own tool registry. ## Redis Key Structure All keys are prefixed with `{keyPrefix}:{swarmId}`: | Key Pattern | Type | Purpose | | ----------------------------------------- | ------------- | ------------------------------- | | `swarm:{id}:messages` | List | Message bus message history | | `swarm:{id}:channel:{target}` | Pub/Sub | Real-time message delivery | | `swarm:{id}:blackboard:{section}` | String (JSON) | Blackboard section data | | `swarm:{id}:blackboard:{section}:history` | List | Blackboard write history | | `swarm:{id}:blackboard:changes` | Pub/Sub | Blackboard change notifications | | `swarm:{id}:events` | List | Event history | | `swarm:{id}:events:live` | Pub/Sub | Live event streaming | | `swarm:{id}:results` | Pub/Sub | Agent job results | | `swarm:jobs:{queue}` | List | Job queue for worker nodes | ## RedisMessageBus Drop-in replacement for `InMemoryMessageBus`. Messages are stored in a Redis list and delivered in real-time via pub/sub pattern subscriptions. ```typescript import { RedisMessageBus } from '@cogitator-ai/swarms'; import Redis from 'ioredis'; const bus = new RedisMessageBus( { enabled: true }, { redis: new Redis(), swarmId: 'swarm_abc123', keyPrefix: 'swarm' } ); await bus.initialize(); await bus.send({ swarmId: 'swarm_abc123', from: 'agent-a', to: 'agent-b', type: 'request', content: 'Need your analysis', }); await bus.syncFromRedis(); await bus.close(); ``` The bus maintains a local cache for fast reads and subscribes to Redis pub/sub channels using pattern matching (`swarm:{id}:channel:*`) for real-time delivery. ## RedisBlackboard Shared state synchronized across all nodes. Each section is stored as a JSON string in Redis, with change notifications broadcast via pub/sub. ```typescript import { RedisBlackboard } from '@cogitator-ai/swarms'; import Redis from 'ioredis'; const bb = new RedisBlackboard( { enabled: true, sections: { tasks: [] }, trackHistory: true }, { redis: new Redis(), swarmId: 'swarm_abc123', keyPrefix: 'swarm' } ); await bb.initialize(); bb.write('tasks', [{ id: 1, title: 'Implement auth' }], 'lead'); const tasks = bb.read('tasks'); bb.subscribe('tasks', (data, agentName) => { console.log(`Tasks updated by ${agentName}`); }); await bb.syncFromRedis(); await bb.close(); ``` On initialization, existing sections are loaded from Redis. The local cache provides instant reads, while writes are persisted to Redis and broadcast to all subscribers. ## RedisSwarmEventEmitter Events are persisted in a Redis list and streamed live to all connected nodes via pub/sub. ```typescript import { RedisSwarmEventEmitter } from '@cogitator-ai/swarms'; import Redis from 'ioredis'; const events = new RedisSwarmEventEmitter({ redis: new Redis(), swarmId: 'swarm_abc123', keyPrefix: 'swarm', maxEvents: 1000, }); await events.initialize(); events.on('agent:complete', (event) => { console.log(`Agent done: ${event.agentName}`); }); await events.emitAsync('agent:start', { agentName: 'worker-a' }); const history = await events.getEventsAsync(); await events.close(); ``` The emitter auto-trims the event list in Redis to `maxEvents` entries to prevent unbounded growth. ## Running Worker Nodes `DistributedSwarmWorker` from `@cogitator-ai/worker` consumes the job queue, runs each agent turn with its own `Cogitator` and tools, and publishes the result back to the coordinator: ```typescript import { Cogitator } from '@cogitator-ai/core'; import { DistributedSwarmWorker } from '@cogitator-ai/worker'; const worker = new DistributedSwarmWorker( { redis: { host: 'redis.example.com', port: 6379, password: process.env.REDIS_PASSWORD }, keyPrefix: 'swarm', // must match distributed.redis.keyPrefix queue: 'swarm-agent-jobs', // must match distributed.queue concurrency: 4, cogitator: new Cogitator({ llm: { defaultModel: 'ollama/llama3.2' } }), tools: [searchTool, calculatorTool], }, { onJobFailed: (job, error) => console.error(job.agentName, error.message), } ); await worker.start(); process.on('SIGTERM', () => void worker.stop()); ``` Agents reference tools by name; a turn that needs a tool missing from the worker's `tools` fails with a clear error instead of running with a fake implementation. `stop()` stops taking new jobs and waits for in-flight turns. ## Scaling Considerations **Horizontal scaling** — spin up multiple worker nodes consuming from the same job queue. Redis `BLPOP` ensures each job is processed by exactly one worker; `concurrency` controls parallel turns per node. **State consistency** — the blackboard and message bus use Redis as the source of truth. Local caches provide fast reads; writes are immediately persisted and broadcast. **Connection management** — a distributed swarm uses a command connection plus subscriber connections for messages, blackboard, events and results. Call `close()` when done to release them. **Timeouts** — configure the `timeout` option based on your expected agent execution time. Jobs that exceed the timeout will cause the coordinator to throw an error. **Key cleanup** — swarm keys persist in Redis after execution. For production use, set a TTL on keys or implement periodic cleanup based on the `swarmId`.