# Document Loaders

URL: https://cogitator.app/docs/rag/loaders

> Load documents from text files, markdown, JSON, CSV, HTML, PDFs, and web URLs into the RAG pipeline.

## Overview

Document loaders convert raw sources into `RAGDocument` objects that the pipeline can chunk and embed. Every loader implements the `DocumentLoader` interface:

```typescript
interface DocumentLoader {
  load(source: string): Promise<RAGDocument[]>;
  readonly supportedTypes: string[];
}
```

The `source` parameter is a file path for file-based loaders, or a URL for `WebLoader`. Directory paths are supported by `TextLoader` and `MarkdownLoader` — they load the matching files directly inside the directory (not subdirectories), in name order. Each document gets a random `id`, `source` (absolute path or URL), `sourceType` and optional `metadata`, which the pipeline copies onto every chunk.

A pipeline uses a single loader; see [Mixing Sources](#mixing-sources) to ingest several file types.

## TextLoader

Loads plain text files (`.txt`, `.text`). Accepts a file path or directory.

```typescript
import { TextLoader } from '@cogitator-ai/rag';

const loader = new TextLoader();

const docs = await loader.load('./data/notes.txt');
const allDocs = await loader.load('./data/text-files/');
```

No options. Documents get `sourceType: 'text'`.

## MarkdownLoader

Loads `.md` and `.mdx` files. Optionally strips YAML frontmatter and extracts it as metadata.

```typescript
import { MarkdownLoader } from '@cogitator-ai/rag';

const loader = new MarkdownLoader({ stripFrontmatter: true });
const docs = await loader.load('./docs/');

console.log(docs[0].metadata);
// { title: 'Getting Started', description: 'Quick start guide' }
```

| Option             | Type    | Default | Description                                      |
| ------------------ | ------- | ------- | ------------------------------------------------ |
| `stripFrontmatter` | boolean | false   | Remove YAML frontmatter and parse it as metadata |

## JSONLoader

Loads JSON files. Handles both single objects and arrays. Automatically detects `content`, `text`, or `body` fields, or you can specify a custom content field.

```typescript
import { JSONLoader } from '@cogitator-ai/rag';

const loader = new JSONLoader({
  contentField: 'description',
  metadataFields: ['category', 'author'],
});

const docs = await loader.load('./data/articles.json');
```

If no content field is found, the entire object is serialized as the document content; non-string field values are serialized as JSON, and array items that are not objects become their string value. `source` must be a single file.

| Option           | Type      | Default | Description                            |
| ---------------- | --------- | ------- | -------------------------------------- |
| `contentField`   | string    | auto    | Field name to use as document content  |
| `metadataFields` | string\[] | —       | Fields to extract as document metadata |

Auto-detected content fields (in order): `content`, `text`, `body`.

## CSVLoader

Loads CSV files. Each row becomes a separate document. Requires `papaparse` as a peer dependency.

```bash
pnpm add papaparse
```

```typescript
import { CSVLoader } from '@cogitator-ai/rag';

const loader = new CSVLoader({
  contentColumn: 'description',
  metadataColumns: ['id', 'category'],
  delimiter: ',',
});

const docs = await loader.load('./data/products.csv');
```

The first row is the header. If no `contentColumn` is specified, the first column is used; a `contentColumn` missing from the header throws. Every document gets `metadata.row` (1-based data row number) plus the `metadataColumns` values.

| Option            | Type      | Default      | Description                       |
| ----------------- | --------- | ------------ | --------------------------------- |
| `contentColumn`   | string    | first column | Column to use as document content |
| `metadataColumns` | string\[] | —            | Columns to extract as metadata    |
| `delimiter`       | string    | auto         | CSV delimiter character           |

## HTMLLoader

Loads HTML files and extracts text content using CSS selectors. Requires `cheerio` as a peer dependency.

```bash
pnpm add cheerio
```

```typescript
import { HTMLLoader } from '@cogitator-ai/rag';

const loader = new HTMLLoader({ selector: 'article' });
const docs = await loader.load('./pages/about.html');

console.log(docs[0].metadata);
// { title: 'About Us' }
```

The `<title>` tag is automatically extracted as metadata if present. When the selector matches several elements, their texts are joined with blank lines. Scripts, styles, `noscript`, `template`, `svg`, `iframe`, `object` and `<head>` content is removed, block elements (paragraphs, headings, list items, table rows…) become line breaks, `<br>` becomes a newline and whitespace is normalized — so `<p>One</p><p>Two</p>` yields `One\n\nTwo` rather than `OneTwo`.

| Option     | Type   | Default | Description                         |
| ---------- | ------ | ------- | ----------------------------------- |
| `selector` | string | `body`  | CSS selector for content extraction |

## PDFLoader

Loads PDF files and extracts text. Optionally splits into one document per page. Requires `pdf-parse` as a peer dependency.

```bash
pnpm add pdf-parse
```

```typescript
import { PDFLoader } from '@cogitator-ai/rag';

const loader = new PDFLoader();
const docs = await loader.load('./papers/research.pdf');
console.log(docs[0].metadata);
// { pages: 42 }

const perPage = new PDFLoader({ splitPages: true });
const pageDocs = await perPage.load('./papers/research.pdf');
console.log(pageDocs[0].metadata);
// { pageNumber: 1, totalPages: 42 }
```

| Option       | Type    | Default | Description                                 |
| ------------ | ------- | ------- | ------------------------------------------- |
| `splitPages` | boolean | false   | Create one document per page instead of one |

When the PDF declares a title, it is added as `metadata.title`. In per-page mode, pages without text are skipped and the remaining documents keep their physical `pageNumber`.

## WebLoader

Fetches a URL and extracts text content. HTML is processed by `HTMLLoader` (requires `cheerio`); `text/*` and JSON responses are kept as raw text; other content types are rejected. Redirects are followed (max 5), responses are decompressed (gzip/deflate/br), decoded with the declared charset and capped at `maxResponseBytes`. Documents get `sourceType: 'web'`, `metadata.url` (final URL after redirects), `metadata.contentType` and, for HTML, `metadata.title`. Custom `headers` are merged over the defaults (`user-agent: cogitator-rag-webloader` and an HTML-first `accept`).

```bash
pnpm add cheerio
```

```typescript
import { WebLoader } from '@cogitator-ai/rag';

const loader = new WebLoader({
  selector: 'main',
  headers: { 'User-Agent': 'CogitatorBot/1.0' },
});

const docs = await loader.load('https://example.com/docs/getting-started');
```

| Option                | Type                    | Default | Description                                        |
| --------------------- | ----------------------- | ------- | -------------------------------------------------- |
| `selector`            | string                  | `body`  | CSS selector for content extraction                |
| `headers`             | Record\<string, string> | —       | Custom HTTP headers for the request                |
| `timeoutMs`           | number                  | `30000` | Per-request timeout                                |
| `maxResponseBytes`    | number                  | 50MB    | Maximum (decompressed) response size               |
| `allowPrivateNetwork` | boolean                 | `false` | Allow loopback/private hosts (e.g. intranet wikis) |
| `robots`              | `RobotsChecker`         | —       | Check robots.txt before every fetch and redirect   |

### SSRF protection

By default `WebLoader` refuses loopback, private (RFC 1918), link-local, CGNAT, multicast and cloud-metadata targets — including IPv4-mapped / NAT64 IPv6 literals, numeric host forms such as `http://2130706433/`, and hostnames that **resolve** to such addresses. Resolution is checked at connect time for every redirect hop, so DNS rebinding cannot bypass the check. Set `allowPrivateNetwork: true` only for trusted URLs.

### robots.txt

Pass a `robots` checker and every URL is checked against its site's robots.txt before it is fetched, redirect targets included. A disallowed URL fails the load with `RobotsDisallowedError` and is never requested. `RobotsPolicy` from `@cogitator-ai/core` reads and caches robots.txt per site by RFC 9309 (longest match wins, `*` and \`---
title: Document Loaders
description: Load documents from text files, markdown, JSON, CSV, HTML, PDFs, and web URLs into the RAG pipeline.

***

## Overview

Document loaders convert raw sources into `RAGDocument` objects that the pipeline can chunk and embed. Every loader implements the `DocumentLoader` interface:

```typescript
interface DocumentLoader {
  load(source: string): Promise<RAGDocument[]>;
  readonly supportedTypes: string[];
}
```

The `source` parameter is a file path for file-based loaders, or a URL for `WebLoader`. Directory paths are supported by `TextLoader` and `MarkdownLoader` — they load the matching files directly inside the directory (not subdirectories), in name order. Each document gets a random `id`, `source` (absolute path or URL), `sourceType` and optional `metadata`, which the pipeline copies onto every chunk.

A pipeline uses a single loader; see [Mixing Sources](#mixing-sources) to ingest several file types.

## TextLoader

Loads plain text files (`.txt`, `.text`). Accepts a file path or directory.

```typescript
import { TextLoader } from '@cogitator-ai/rag';

const loader = new TextLoader();

const docs = await loader.load('./data/notes.txt');
const allDocs = await loader.load('./data/text-files/');
```

No options. Documents get `sourceType: 'text'`.

## MarkdownLoader

Loads `.md` and `.mdx` files. Optionally strips YAML frontmatter and extracts it as metadata.

```typescript
import { MarkdownLoader } from '@cogitator-ai/rag';

const loader = new MarkdownLoader({ stripFrontmatter: true });
const docs = await loader.load('./docs/');

console.log(docs[0].metadata);
// { title: 'Getting Started', description: 'Quick start guide' }
```

| Option             | Type    | Default | Description                                      |
| ------------------ | ------- | ------- | ------------------------------------------------ |
| `stripFrontmatter` | boolean | false   | Remove YAML frontmatter and parse it as metadata |

## JSONLoader

Loads JSON files. Handles both single objects and arrays. Automatically detects `content`, `text`, or `body` fields, or you can specify a custom content field.

```typescript
import { JSONLoader } from '@cogitator-ai/rag';

const loader = new JSONLoader({
  contentField: 'description',
  metadataFields: ['category', 'author'],
});

const docs = await loader.load('./data/articles.json');
```

If no content field is found, the entire object is serialized as the document content; non-string field values are serialized as JSON, and array items that are not objects become their string value. `source` must be a single file.

| Option           | Type      | Default | Description                            |
| ---------------- | --------- | ------- | -------------------------------------- |
| `contentField`   | string    | auto    | Field name to use as document content  |
| `metadataFields` | string\[] | —       | Fields to extract as document metadata |

Auto-detected content fields (in order): `content`, `text`, `body`.

## CSVLoader

Loads CSV files. Each row becomes a separate document. Requires `papaparse` as a peer dependency.

```bash
pnpm add papaparse
```

```typescript
import { CSVLoader } from '@cogitator-ai/rag';

const loader = new CSVLoader({
  contentColumn: 'description',
  metadataColumns: ['id', 'category'],
  delimiter: ',',
});

const docs = await loader.load('./data/products.csv');
```

The first row is the header. If no `contentColumn` is specified, the first column is used; a `contentColumn` missing from the header throws. Every document gets `metadata.row` (1-based data row number) plus the `metadataColumns` values.

| Option            | Type      | Default      | Description                       |
| ----------------- | --------- | ------------ | --------------------------------- |
| `contentColumn`   | string    | first column | Column to use as document content |
| `metadataColumns` | string\[] | —            | Columns to extract as metadata    |
| `delimiter`       | string    | auto         | CSV delimiter character           |

## HTMLLoader

Loads HTML files and extracts text content using CSS selectors. Requires `cheerio` as a peer dependency.

```bash
pnpm add cheerio
```

```typescript
import { HTMLLoader } from '@cogitator-ai/rag';

const loader = new HTMLLoader({ selector: 'article' });
const docs = await loader.load('./pages/about.html');

console.log(docs[0].metadata);
// { title: 'About Us' }
```

The `<title>` tag is automatically extracted as metadata if present. When the selector matches several elements, their texts are joined with blank lines. Scripts, styles, `noscript`, `template`, `svg`, `iframe`, `object` and `<head>` content is removed, block elements (paragraphs, headings, list items, table rows…) become line breaks, `<br>` becomes a newline and whitespace is normalized — so `<p>One</p><p>Two</p>` yields `One\n\nTwo` rather than `OneTwo`.

| Option     | Type   | Default | Description                         |
| ---------- | ------ | ------- | ----------------------------------- |
| `selector` | string | `body`  | CSS selector for content extraction |

## PDFLoader

Loads PDF files and extracts text. Optionally splits into one document per page. Requires `pdf-parse` as a peer dependency.

```bash
pnpm add pdf-parse
```

```typescript
import { PDFLoader } from '@cogitator-ai/rag';

const loader = new PDFLoader();
const docs = await loader.load('./papers/research.pdf');
console.log(docs[0].metadata);
// { pages: 42 }

const perPage = new PDFLoader({ splitPages: true });
const pageDocs = await perPage.load('./papers/research.pdf');
console.log(pageDocs[0].metadata);
// { pageNumber: 1, totalPages: 42 }
```

| Option       | Type    | Default | Description                                 |
| ------------ | ------- | ------- | ------------------------------------------- |
| `splitPages` | boolean | false   | Create one document per page instead of one |

When the PDF declares a title, it is added as `metadata.title`. In per-page mode, pages without text are skipped and the remaining documents keep their physical `pageNumber`.

## WebLoader

Fetches a URL and extracts text content. HTML is processed by `HTMLLoader` (requires `cheerio`); `text/*` and JSON responses are kept as raw text; other content types are rejected. Redirects are followed (max 5), responses are decompressed (gzip/deflate/br), decoded with the declared charset and capped at `maxResponseBytes`. Documents get `sourceType: 'web'`, `metadata.url` (final URL after redirects), `metadata.contentType` and, for HTML, `metadata.title`. Custom `headers` are merged over the defaults (`user-agent: cogitator-rag-webloader` and an HTML-first `accept`).

```bash
pnpm add cheerio
```

```typescript
import { WebLoader } from '@cogitator-ai/rag';

const loader = new WebLoader({
  selector: 'main',
  headers: { 'User-Agent': 'CogitatorBot/1.0' },
});

const docs = await loader.load('https://example.com/docs/getting-started');
```

| Option                | Type                    | Default | Description                                        |
| --------------------- | ----------------------- | ------- | -------------------------------------------------- |
| `selector`            | string                  | `body`  | CSS selector for content extraction                |
| `headers`             | Record\<string, string> | —       | Custom HTTP headers for the request                |
| `timeoutMs`           | number                  | `30000` | Per-request timeout                                |
| `maxResponseBytes`    | number                  | 50MB    | Maximum (decompressed) response size               |
| `allowPrivateNetwork` | boolean                 | `false` | Allow loopback/private hosts (e.g. intranet wikis) |
| `robots`              | `RobotsChecker`         | —       | Check robots.txt before every fetch and redirect   |

### SSRF protection

By default `WebLoader` refuses loopback, private (RFC 1918), link-local, CGNAT, multicast and cloud-metadata targets — including IPv4-mapped / NAT64 IPv6 literals, numeric host forms such as `http://2130706433/`, and hostnames that **resolve** to such addresses. Resolution is checked at connect time for every redirect hop, so DNS rebinding cannot bypass the check. Set `allowPrivateNetwork: true` only for trusted URLs.

patterns, a 4xx robots.txt allows everything, a 5xx or a network failure allows nothing until the cache expires):

```typescript
import { RobotsPolicy } from '@cogitator-ai/core';
import { WebLoader } from '@cogitator-ai/rag';

const userAgent = 'NewsBot/1.0 (+https://example.com/bot)';
const loader = new WebLoader({
  headers: { 'user-agent': userAgent },
  robots: new RobotsPolicy({ userAgent }), // matches "User-agent: NewsBot" groups, else "*"
});
```

## Custom Loaders

Implement the `DocumentLoader` interface to load from any source:

```typescript
import { randomUUID } from 'node:crypto';
import type { DocumentLoader, RAGDocument } from '@cogitator-ai/rag';

class NotionLoader implements DocumentLoader {
  readonly supportedTypes = ['notion'];

  constructor(private readonly fetchPage: (pageId: string) => Promise<string>) {}

  async load(pageId: string): Promise<RAGDocument[]> {
    const content = await this.fetchPage(pageId);

    return [
      {
        id: randomUUID(),
        content,
        source: `notion://${pageId}`,
        sourceType: 'text',
        metadata: { pageId },
      },
    ];
  }
}
```

`sourceType` must be one of `'pdf' | 'html' | 'csv' | 'json' | 'markdown' | 'text' | 'web'`.

Pass your custom loader to the pipeline builder:

```typescript
const pipeline = new RAGPipelineBuilder()
  .withLoader(new NotionLoader(fetchNotionPage))
  // ...
  .build();

await pipeline.ingest('page-id-here');
```

## Mixing Sources

To ingest several file types through one pipeline, wrap the built-in loaders in a loader that picks one by extension:

```typescript
import { extname } from 'node:path';
import {
  MarkdownLoader,
  PDFLoader,
  TextLoader,
  WebLoader,
  type DocumentLoader,
  type RAGDocument,
} from '@cogitator-ai/rag';

const byExtension: Record<string, DocumentLoader> = {
  '.md': new MarkdownLoader({ stripFrontmatter: true }),
  '.mdx': new MarkdownLoader({ stripFrontmatter: true }),
  '.pdf': new PDFLoader({ splitPages: true }),
  '.txt': new TextLoader(),
};
const web = new WebLoader();

const mixedLoader: DocumentLoader = {
  supportedTypes: ['md', 'mdx', 'pdf', 'txt', 'http', 'https'],
  async load(source: string): Promise<RAGDocument[]> {
    if (/^https?:\/\//.test(source)) return web.load(source);
    const loader = byExtension[extname(source).toLowerCase()];
    if (!loader) throw new Error(`No loader for ${source}`);
    return loader.load(source);
  },
};
```