# Built-in Tools

URL: https://cogitator.app/docs/tools/built-in

> Reference for the 26 built-in tools that ship with @cogitator-ai/core, the environment variables they read, and the tool factories next to them.

## Overview

Cogitator ships with **26 built-in tools** covering math, data processing, the file system, network, databases, email and GitHub. Every tool has a Zod schema (the runtime validates the model's arguments against it) and returns structured results; most failures come back as `{ error }` data the model can read instead of a thrown exception.

```typescript
import { Agent, builtinTools, calculator, fileRead, httpRequest } from '@cogitator-ai/core';

const assistant = new Agent({
  name: 'assistant',
  model: 'openai/gpt-6.1-sol',
  instructions: 'You are a helpful assistant.',
  tools: [calculator, fileRead, httpRequest],
});

const everything = new Agent({
  name: 'power-user',
  model: 'openai/gpt-6.1-sol',
  instructions: 'You are a helpful assistant.',
  tools: builtinTools,
});
```

`builtinTools` is a `Tool[]`, so pass it as `tools` directly or spread it next to your own tools. Prefer picking the tools an agent needs: `builtinTools` includes `exec`, `file_delete` and `send_email`.

| Export          | Tool name        | Export         | Tool name       |
| --------------- | ---------------- | -------------- | --------------- |
| `calculator`    | `calculator`     | `fileRead`     | `file_read`     |
| `datetime`      | `datetime`       | `fileWrite`    | `file_write`    |
| `uuid`          | `uuid`           | `fileList`     | `file_list`     |
| `randomNumber`  | `random_number`  | `fileExists`   | `file_exists`   |
| `randomString`  | `random_string`  | `fileDelete`   | `file_delete`   |
| `hash`          | `hash`           | `httpRequest`  | `http_request`  |
| `base64Encode`  | `base64_encode`  | `exec`         | `exec`          |
| `base64Decode`  | `base64_decode`  | `webSearch`    | `web_search`    |
| `sleep`         | `sleep`          | `webScrape`    | `web_scrape`    |
| `jsonParse`     | `json_parse`     | `sqlQuery`     | `sql_query`     |
| `jsonStringify` | `json_stringify` | `vectorSearch` | `vector_search` |
| `regexMatch`    | `regex_match`    | `sendEmail`    | `send_email`    |
| `regexReplace`  | `regex_replace`  | `githubApi`    | `github_api`    |

### Environment Variables

| Variable                                                                             | Read by                                                           |
| ------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| `TAVILY_API_KEY`, `BRAVE_API_KEY`, `SERPER_API_KEY`                                  | `web_search` (first one set wins when `provider` is omitted)      |
| `DATABASE_URL`                                                                       | `sql_query`, `vector_search` when no `connectionString` is passed |
| `OPENAI_API_KEY`, `OLLAMA_BASE_URL` / `OLLAMA_URL` / `OLLAMA_HOST`, `GOOGLE_API_KEY` | `vector_search` embeddings (auto-detected in that order)          |
| `RESEND_API_KEY`, `RESEND_FROM_EMAIL`                                                | `send_email` via Resend                                           |
| `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASS`, `SMTP_FROM`                      | `send_email` via SMTP                                             |
| `GITHUB_TOKEN`                                                                       | `github_api`                                                      |

Variables are read on each call. On runtimes without `process.env` (Cloudflare Workers without `nodejs_compat`, Deno without `--allow-env`) they count as unset.

### Optional Packages

| Package          | Needed by                                  |
| ---------------- | ------------------------------------------ |
| `pg`             | `sql_query` on PostgreSQL, `vector_search` |
| `better-sqlite3` | `sql_query` on SQLite                      |
| `nodemailer`     | `send_email` via SMTP                      |

They are optional peer dependencies of `@cogitator-ai/core`; a tool that needs a missing one returns an error telling you what to install.

***

## Math & Utilities

### `calculator`

Evaluates mathematical expressions without `eval`. Supports `+`, `-`, `*`, `/`, `^` (power), parentheses, and functions: `sqrt`, `sin`, `cos`, `tan`, `log`, `abs`, `round`, `floor`, `ceil`. Constants: `pi`, `e`.

| Parameter    | Type     | Description                         |
| ------------ | -------- | ----------------------------------- |
| `expression` | `string` | Math expression, e.g. `"2 + 3 * 4"` |

### `uuid`

Generates UUID v4 identifiers with `crypto.randomUUID`.

| Parameter | Type     | Description                                      |
| --------- | -------- | ------------------------------------------------ |
| `count`   | `number` | Number of UUIDs to generate (1 - 100, default 1) |

### `random_number`

Generates a random number within a range.

| Parameter | Type      | Description                     |
| --------- | --------- | ------------------------------- |
| `min`     | `number`  | Minimum value (default: 0)      |
| `max`     | `number`  | Maximum value (default: 1)      |
| `integer` | `boolean` | Return integer (default: false) |

### `random_string`

Generates a cryptographically secure random string using `crypto.randomBytes`.

| Parameter | Type     | Description                                                             |
| --------- | -------- | ----------------------------------------------------------------------- |
| `length`  | `number` | String length (1 - 1000)                                                |
| `charset` | `enum`   | `alphanumeric` \| `alpha` \| `numeric` \| `hex` (default: alphanumeric) |

### `hash`

Computes cryptographic hashes.

| Parameter   | Type     | Description                                               |
| ----------- | -------- | --------------------------------------------------------- |
| `data`      | `string` | String to hash                                            |
| `algorithm` | `enum`   | `md5` \| `sha1` \| `sha256` \| `sha512` (default: sha256) |
| `encoding`  | `enum`   | `hex` \| `base64` (default: hex)                          |

### `sleep`

Pauses execution for a specified duration. Maximum 60 seconds.

| Parameter | Type     | Description                       |
| --------- | -------- | --------------------------------- |
| `ms`      | `number` | Milliseconds to sleep (0 - 60000) |

### `datetime`

Returns the current date and time in various formats and timezones.

| Parameter  | Type     | Description                                                      |
| ---------- | -------- | ---------------------------------------------------------------- |
| `timezone` | `string` | IANA timezone, e.g. `"America/New_York"` (default: system zone)  |
| `format`   | `enum`   | `iso` \| `unix` \| `readable` \| `date` \| `time` (default: iso) |

***

## Data Processing

### `json_parse`

Parses a JSON string into an object. Returns `{ result, valid: true }` on success, `{ error, valid: false }` on failure.

| Parameter | Type     | Description          |
| --------- | -------- | -------------------- |
| `json`    | `string` | JSON string to parse |

### `json_stringify`

Converts a value to a JSON string with optional pretty-printing.

| Parameter | Type      | Description                          |
| --------- | --------- | ------------------------------------ |
| `data`    | `unknown` | Data to serialize                    |
| `pretty`  | `boolean` | Enable indentation (default: false)  |
| `indent`  | `number`  | Spaces for indent (0 - 8, default 2) |

### `base64_encode` / `base64_decode`

Encode strings to Base64 or decode Base64 back to plain text. Both support the URL-safe variant.

| Parameter | Type      | Description                              |
| --------- | --------- | ---------------------------------------- |
| `data`    | `string`  | Input string                             |
| `urlSafe` | `boolean` | URL-safe Base64 variant (default: false) |

### `regex_match`

Finds matches of a regular expression in text, returning match strings, indices, and named groups.

| Parameter | Type     | Description                  |
| --------- | -------- | ---------------------------- |
| `text`    | `string` | Text to search               |
| `pattern` | `string` | Regular expression pattern   |
| `flags`   | `string` | Regex flags (default: `"g"`) |

### `regex_replace`

Replaces regex matches in text. Supports `$1`, `$2` capture group references.

| Parameter     | Type     | Description                  |
| ------------- | -------- | ---------------------------- |
| `text`        | `string` | Input text                   |
| `pattern`     | `string` | Regex pattern                |
| `replacement` | `string` | Replacement string           |
| `flags`       | `string` | Regex flags (default: `"g"`) |

Both regex tools reject texts over 1,000,000 characters and stop matching after 5 seconds. They run on the host event loop; for untrusted patterns prefer the WASM [`regex`](/docs/tools/wasm-tools) tool, which runs in a worker that the timeout terminates.

***

## File System

`file_write` and `file_delete` declare `sideEffects: ['filesystem']`. None of the file tools restricts paths — they act with the permissions of your process, so give them only to agents you trust or put the agent behind [approvals](/docs/tools/approvals).

### `file_read`

Reads file contents. Use `base64` encoding for binary files.

| Parameter  | Type     | Description                          |
| ---------- | -------- | ------------------------------------ |
| `path`     | `string` | Path to the file                     |
| `encoding` | `enum`   | `utf-8` \| `base64` (default: utf-8) |

### `file_write`

Writes content to a file. Creates parent directories by default.

| Parameter    | Type      | Description                          |
| ------------ | --------- | ------------------------------------ |
| `path`       | `string`  | File path                            |
| `content`    | `string`  | Content to write                     |
| `encoding`   | `enum`    | `utf-8` \| `base64` (default: utf-8) |
| `createDirs` | `boolean` | Create parent dirs (default: true)   |

### `file_list`

Lists directory contents with file names, paths, types, and sizes.

| Parameter       | Type      | Description                                  |
| --------------- | --------- | -------------------------------------------- |
| `path`          | `string`  | Directory path                               |
| `recursive`     | `boolean` | Recurse into subdirectories (default: false) |
| `includeHidden` | `boolean` | Include dotfiles (default: false)            |

### `file_exists`

Checks if a file or directory exists and returns its type and size.

| Parameter | Type     | Description   |
| --------- | -------- | ------------- |
| `path`    | `string` | Path to check |

### `file_delete`

Deletes a file or directory. Use `recursive: true` for non-empty directories.

| Parameter   | Type      | Description                                  |
| ----------- | --------- | -------------------------------------------- |
| `path`      | `string`  | Path to delete                               |
| `recursive` | `boolean` | Delete contents recursively (default: false) |

***

## Network & Web

### `http_request`

Makes HTTP requests with custom headers, body, and timeout. Response bodies longer than 100,000 characters are truncated (`truncated: true`).

| Parameter | Type                     | Description                                                                           |
| --------- | ------------------------ | ------------------------------------------------------------------------------------- |
| `url`     | `string`                 | Request URL                                                                           |
| `method`  | `enum`                   | `GET` \| `POST` \| `PUT` \| `PATCH` \| `DELETE` \| `HEAD` \| `OPTIONS` (default: GET) |
| `headers` | `Record<string, string>` | Request headers                                                                       |
| `body`    | `string`                 | Request body (for POST/PUT/PATCH)                                                     |
| `timeout` | `number`                 | Timeout in ms (1000 - 60000, default: 30000)                                          |

### `web_search`

Searches the web via Tavily, Brave Search, or Serper. Without `provider` it uses the first of `TAVILY_API_KEY`, `BRAVE_API_KEY`, `SERPER_API_KEY` that is set.

| Parameter           | Type                            | Description                                                                     |
| ------------------- | ------------------------------- | ------------------------------------------------------------------------------- |
| `query`             | `string`                        | Search query                                                                    |
| `provider`          | `enum`                          | `tavily` \| `brave` \| `serper`                                                 |
| `maxResults`        | `number`                        | Max results (1 - 20, default: 5)                                                |
| `page`              | `number`                        | Page of results (1 - 10, Brave and Serper only)                                 |
| `topic`             | `enum`                          | `general` \| `news` (default: general)                                          |
| `recency`           | `enum`                          | `day` \| `week` \| `month` \| `year`: only results from that period             |
| `dateRange`         | `{ from: string, to?: string }` | Only results published within these days (`YYYY-MM-DD`, `to` defaults to today) |
| `includeDomains`    | `string[]`                      | Search only these domains, e.g. `["reuters.com"]`                               |
| `excludeDomains`    | `string[]`                      | Leave out results from these domains                                            |
| `country`           | `string`                        | Favour results from a country, ISO 3166-1 alpha-2 (`us`, `de`)                  |
| `language`          | `string`                        | Favour results in a language, ISO 639-1 (`en`, `pt-br`)                         |
| `searchDepth`       | `enum`                          | `ultra-fast` \| `fast` \| `basic` \| `advanced` (Tavily only)                   |
| `includeAnswer`     | `boolean`                       | Include AI summary (Tavily only)                                                |
| `includeRawContent` | `boolean`                       | Full page content as markdown in each result (Tavily only)                      |

The filters work on every provider, each through its own parameters: Tavily's `topic`, `time_range`, `start_date`, `include_domains`, `country` and `language`, the news endpoints, `freshness`, `country` and `search_lang` of Brave, and the news endpoint, `tbs`, `gl` and `hl` of Serper. Domain filters become `site:` operators for Brave and Serper. A filter a provider cannot apply comes back as an `error` instead of being dropped: Tavily has no result pages, and it favours a country only in general searches.

Results carry `publishedAt` (ISO 8601) when the provider reports a date, Serper's relative dates ("3 hours ago") resolved at the time of the search, `source` for Serper news, and `content` with `includeRawContent`.

`createWebSearchTool({ provider, apiKeys })` builds the same tool with a default provider and keys passed in code instead of the environment:

```typescript
import { createWebSearchTool } from '@cogitator-ai/core';

const search = createWebSearchTool({ provider: 'tavily', apiKeys: { tavily: config.tavilyKey } });
```

### `web_scrape`

Fetches an HTML page and extracts its content as text, markdown or raw HTML, optionally limited to a CSS selector, with link and image collection.

| Parameter       | Type      | Description                                          |
| --------------- | --------- | ---------------------------------------------------- |
| `url`           | `string`  | URL to scrape                                        |
| `selector`      | `string`  | CSS selector to extract (e.g. `"article"`)           |
| `format`        | `enum`    | `text` \| `markdown` \| `html` (default: text)       |
| `maxLength`     | `number`  | Max content length (100 - 100000, default: 50000)    |
| `timeout`       | `number`  | Request timeout in ms (1000 - 60000, default: 30000) |
| `includeLinks`  | `boolean` | Extract links (default: false)                       |
| `includeImages` | `boolean` | Extract image URLs (default: false)                  |

Non-HTML responses return an error.

***

## Database & Search

### `sql_query`

Executes SQL queries against PostgreSQL (`pg`) or SQLite (`better-sqlite3`). Supports parameterized queries and is read-only by default.

| Parameter          | Type        | Description                                                       |
| ------------------ | ----------- | ----------------------------------------------------------------- |
| `query`            | `string`    | SQL query                                                         |
| `database`         | `enum`      | `postgres` \| `sqlite` (auto-detected from the connection string) |
| `connectionString` | `string`    | Postgres URL or SQLite file path (defaults to `DATABASE_URL`)     |
| `params`           | `unknown[]` | Parameterized query values                                        |
| `maxRows`          | `number`    | Max rows returned (1 - 1000, default: 100)                        |
| `readOnly`         | `boolean`   | Only allow read queries (default: true)                           |

In read-only mode the query must be a single `SELECT`/`WITH`/`SHOW`/`DESCRIBE`/`EXPLAIN` statement. On PostgreSQL it additionally runs inside a `BEGIN TRANSACTION READ ONLY` block that is always rolled back, so side-effecting functions (for example `nextval()`) are rejected by the database itself; SQLite databases are opened with `readonly: true`. With `readOnly: false`, statements that return no rows (`INSERT`, `UPDATE`, `DELETE`) return `rows: []` and the number of affected rows in `rowCount`.

### `vector_search`

Performs semantic similarity search with pgvector: embeds the query via OpenAI, Ollama, or Google, then finds similar rows in PostgreSQL.

| Parameter           | Type     | Description                                 |
| ------------------- | -------- | ------------------------------------------- |
| `query`             | `string` | Search query (converted to embedding)       |
| `collection`        | `string` | Table name (default: `"documents"`)         |
| `topK`              | `number` | Number of results (1 - 100, default: 5)     |
| `threshold`         | `number` | Minimum similarity 0 - 1 (default: 0.7)     |
| `filter`            | `Record` | Metadata filter as key-value pairs          |
| `embeddingProvider` | `enum`   | `openai` \| `ollama` \| `google`            |
| `embeddingModel`    | `string` | Embedding model override                    |
| `connectionString`  | `string` | PostgreSQL URL (defaults to `DATABASE_URL`) |

The provider is auto-detected from `OPENAI_API_KEY`, then `OLLAMA_BASE_URL`/`OLLAMA_URL`/`OLLAMA_HOST` (the first one set is also the Ollama URL; default `http://localhost:11434`), then `GOOGLE_API_KEY`. Default models: `text-embedding-3-small` (OpenAI), `nomic-embed-text` (Ollama), `gemini-embedding-001` (Google). The embedding dimension must match your pgvector column.

***

## Communication & Dev

### `send_email`

Sends email via the Resend API or SMTP (`nodemailer`). Declares `sideEffects: ['network', 'external']`.

| Parameter  | Type                 | Description                                         |
| ---------- | -------------------- | --------------------------------------------------- |
| `to`       | `string \| string[]` | Recipient(s)                                        |
| `subject`  | `string`             | Email subject                                       |
| `body`     | `string`             | Email body (plain text or HTML)                     |
| `html`     | `boolean`            | Treat body as HTML (default: false)                 |
| `from`     | `string`             | Sender (default: `RESEND_FROM_EMAIL` / `SMTP_FROM`) |
| `replyTo`  | `string`             | Reply-to address                                    |
| `cc`       | `string \| string[]` | CC recipients                                       |
| `bcc`      | `string \| string[]` | BCC recipients                                      |
| `provider` | `enum`               | `resend` \| `smtp` (auto-detected)                  |

Auto-detection picks Resend when `RESEND_API_KEY` is set, otherwise SMTP when `SMTP_HOST` and `SMTP_USER` are set. Resend sends from `onboarding@resend.dev` when no sender is configured. SMTP needs `SMTP_PASS` too, uses port `SMTP_PORT` (default 587, TLS on 465) and falls back to `SMTP_USER` as sender.

### `github_api`

GitHub REST API access with `GITHUB_TOKEN`. One tool, 12 actions: `get_repo`, `list_issues`, `get_issue`, `create_issue`, `update_issue`, `list_prs`, `get_pr`, `create_pr`, `get_file`, `list_commits`, `search_code`, `search_issues`.

| Parameter             | Type       | Description                                      |
| --------------------- | ---------- | ------------------------------------------------ |
| `action`              | `enum`     | One of the actions above                         |
| `owner`, `repo`       | `string`   | Repository                                       |
| `number`              | `number`   | Issue or PR number (get/update)                  |
| `title`, `body`       | `string`   | For new issues and PRs                           |
| `state`               | `enum`     | `open` \| `closed` \| `all` (default: open)      |
| `labels`, `assignees` | `string[]` | For issues                                       |
| `path`, `ref`         | `string`   | File path and ref for `get_file`                 |
| `base`, `head`        | `string`   | Branches for `create_pr`                         |
| `query`               | `string`   | For the search actions                           |
| `perPage`, `page`     | `number`   | Pagination (1 - 100, default 30; default page 1) |

```typescript
import { Agent, githubApi } from '@cogitator-ai/core';

const agent = new Agent({
  name: 'github-bot',
  model: 'openai/gpt-6.1-sol',
  tools: [githubApi],
  instructions: 'You help manage GitHub repositories.',
});
```

### `exec`

Executes shell commands and returns stdout, stderr and the exit code. Declares `sideEffects: ['process', 'filesystem', 'network']` and `requiresApproval: true`, so every call waits for a decision (see [Approvals](/docs/tools/approvals)).

| Parameter | Type     | Description                                  |
| --------- | -------- | -------------------------------------------- |
| `command` | `string` | Shell command to execute                     |
| `cwd`     | `string` | Working directory                            |
| `timeout` | `number` | Timeout in ms (100 - 300000, default: 30000) |
| `env`     | `Record` | Additional environment variables             |

`exec` declares a Docker sandbox (`cogitator/sandbox:base`, 256 MB, 100 PIDs, no network). Inside a `Cogitator` the command runs in that container through `@cogitator-ai/sandbox`. When the sandbox package is not installed or Docker is unreachable, the runtime logs a warning and runs the command **on the host**, unless `sandbox.allowNativeFallback: false` on the `Cogitator` makes the call fail. See [Sandbox](/docs/deployment/sandbox).

***

## Multimodal Tool Factories

Image and audio tools are created with factories rather than shipped in `builtinTools`. See [Vision & Audio](/docs/core/vision-and-audio) for parameters and results.

| Factory                     | Tool              | Default model                                                                              |
| --------------------------- | ----------------- | ------------------------------------------------------------------------------------------ |
| `createAnalyzeImageTool`    | `analyzeImage`    | `gpt-6.1-sol` on the `llm` backend you pass in                                             |
| `createGenerateImageTool`   | `generateImage`   | `gpt-image-2.5-flare`; returns base64 `imageBase64`, `size` / `quality` default to `auto`  |
| `createTranscribeAudioTool` | `transcribeAudio` | `gpt-transcribe`; `whisper-1` when word timestamps are requested without an explicit model |
| `createGenerateSpeechTool`  | `generateSpeech`  | `gpt-4o-mini-tts`                                                                          |

## Other Tool Factories

These build tools around state you provide; they are not part of `builtinTools`.

| Factory                                                                  | Tools                                                            | Notes                                                                                                |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `createMemoryTools({ graphAdapter, agentId, coreFacts?, embeddingFn? })` | `remember`, `recall`, `forget`                                   | Long-term memory in a knowledge graph — see [Smart Memory](/docs/channels/smart-memory)              |
| `createSchedulerTools({ store, defaultChannel?, defaultUserId? })`       | `schedule_task`, `list_tasks`, `cancel_task`                     | Delayed, dated or cron tasks in a workflows `TimerStore` — see [Scheduler](/docs/channels/scheduler) |
| `createCapabilitiesTool(doc)`                                            | `lookup_capabilities`                                            | Searches a capabilities document line by line                                                        |
| `createDeviceTools()`                                                    | `clipboard_read`, `clipboard_write`, `system_info`, `screenshot` | Clipboard via `pbpaste`/`pbcopy` (macOS) or `xclip`; `screenshot` is macOS only                      |
| `createSelfTools({ toolsDir, onToolsChanged? })`                         | `create_tool`, `test_tool`, `list_custom_tools`, `delete_tool`   | Lets the agent write `.mjs` tools into `toolsDir`; `create_tool` requires approval                   |
| `loadCustomTools(toolsDir)`                                              | the tools in `toolsDir`                                          | Loads the `.mjs` default exports `createSelfTools` wrote                                             |

Tools created with `createSelfTools` run with full Node.js access in your process.

## Public hosts only

A model chooses the URLs `web_scrape` and `http_request` fetch, and the audio URL `transcribe_audio` reads, so a page it read or a prompt it was given can point it at `http://169.254.169.254/` (a cloud's credentials), `http://localhost:8080/admin` or a database on the internal network. These tools reach only the public internet: a URL on a loopback, private, link-local, carrier-grade NAT, multicast or documentation address, `localhost`, `*.local`, `*.internal` or a metadata host is refused, every redirect target is checked again, and a hostname is checked by the address it resolves to when the request connects, so a name that resolves elsewhere a moment later (DNS rebinding) is refused too. A refused URL comes back to the model as an `error` naming it.

An agent meant to work on a trusted network opts in, or brings its own client:

```typescript
import { createHttpRequestTool, createWebScrapeTool } from '@cogitator-ai/core';

const intranet = createWebScrapeTool({ allowPrivateNetwork: true });
const api = createHttpRequestTool({ fetch: proxiedFetch });
```

A `fetch` of your own is trusted with every URL, so it should apply your network policy. `createTranscribeAudioTool` takes the same two options.

The guard is yours to use too: `fetchPublic` is a `fetch` for public hosts, `createPublicFetch({ allowPrivateNetwork, maxRedirects, resolver, isBlocked })` builds one to your rules, `assertPublicUrl(url)` checks a URL up front, `assertPublicHost(url)` checks the addresses its host resolves to as well and `isPrivateAddress(ip)` classifies an address. They run on `node:http`, in Node and Bun alike, and a refusal throws a `PrivateNetworkError`.

With `robots`, the tool checks a URL and the addresses its host resolves to before it reads the site's robots.txt. `RobotsPolicy` reads robots.txt with the global `fetch` unless given another: `new RobotsPolicy({ userAgent, fetch: fetchPublic })` keeps those reads on public hosts too.

Requests carry `DEFAULT_USER_AGENT` and `Accept: */*` unless the caller sets its own headers.

## Polite crawling with robots.txt

`RobotsPolicy` reads and caches a site's robots.txt and answers whether a URL may be fetched, by RFC 9309: the groups that name your product token (else `*`), longest match wins with `*` and `$` patterns, a 4xx robots.txt allows everything, a 5xx or a network failure allows nothing until the cache (one hour by default) expires. It implements `RobotsChecker`, which `WebLoader` in `@cogitator-ai/rag` and `BrowserSession` in `@cogitator-ai/browser` take as `robots`:

```typescript
import { RobotsPolicy } from '@cogitator-ai/core';

const robots = new RobotsPolicy({ userAgent: 'NewsBot/1.0 (+https://example.com/bot)' });
await robots.allows('https://example.com/news/today'); // true or false
```

The token defaults to the User-Agent's first word (`NewsBot`); pass `token` to match another. `robotsRulesFor(text, token)` and `robotsAllowsPath(rules, path)` expose the parser for robots.txt files you already have.

`createWebScrapeTool({ userAgent, robots })` builds a `web_scrape` tool with your User-Agent and a robots.txt check on every hop, redirects included (a disallowed page comes back as an `error`). The plain `webScrape` keeps the default User-Agent and checks nothing.