Document Loaders

OverviewCopy Anchor Link

Document loaders convert raw sources into RAGDocument objects that the pipeline can chunk and embed. Every loader implements the DocumentLoader interface:

interface DocumentLoader {
  load(source: string): Promise<RAGDocument[]>;
  readonly supportedTypes: string[];
}

The source parameter is a file path for file-based loaders, or a URL for WebLoader. Directory paths are supported by TextLoader and MarkdownLoader — they load the matching files directly inside the directory (not subdirectories), in name order. Each document gets a random id, source (absolute path or URL), sourceType and optional metadata, which the pipeline copies onto every chunk.

A pipeline uses a single loader; see Mixing Sources to ingest several file types.

TextLoaderCopy Anchor Link

Loads plain text files (.txt, .text). Accepts a file path or directory.

import { TextLoader } from '@cogitator-ai/rag';

const loader = new TextLoader();

const docs = await loader.load('./data/notes.txt');
const allDocs = await loader.load('./data/text-files/');

No options. Documents get sourceType: 'text'.

MarkdownLoaderCopy Anchor Link

Loads .md and .mdx files. Optionally strips YAML frontmatter and extracts it as metadata.

import { MarkdownLoader } from '@cogitator-ai/rag';

const loader = new MarkdownLoader({ stripFrontmatter: true });
const docs = await loader.load('./docs/');

console.log(docs[0].metadata);
// { title: 'Getting Started', description: 'Quick start guide' }

JSONLoaderCopy Anchor Link

Loads JSON files. Handles both single objects and arrays. Automatically detects content, text, or body fields, or you can specify a custom content field.

import { JSONLoader } from '@cogitator-ai/rag';

const loader = new JSONLoader({
  contentField: 'description',
  metadataFields: ['category', 'author'],
});

const docs = await loader.load('./data/articles.json');

If no content field is found, the entire object is serialized as the document content; non-string field values are serialized as JSON, and array items that are not objects become their string value. source must be a single file.

Auto-detected content fields (in order): content, text, body.

CSVLoaderCopy Anchor Link

Loads CSV files. Each row becomes a separate document. Requires papaparse as a peer dependency.

pnpm add papaparse

import { CSVLoader } from '@cogitator-ai/rag';

const loader = new CSVLoader({
  contentColumn: 'description',
  metadataColumns: ['id', 'category'],
  delimiter: ',',
});

const docs = await loader.load('./data/products.csv');

The first row is the header. If no contentColumn is specified, the first column is used; a contentColumn missing from the header throws. Every document gets metadata.row (1-based data row number) plus the metadataColumns values.

HTMLLoaderCopy Anchor Link

Loads HTML files and extracts text content using CSS selectors. Requires cheerio as a peer dependency.

pnpm add cheerio

import { HTMLLoader } from '@cogitator-ai/rag';

const loader = new HTMLLoader({ selector: 'article' });
const docs = await loader.load('./pages/about.html');

console.log(docs[0].metadata);
// { title: 'About Us' }

The <title> tag is automatically extracted as metadata if present. When the selector matches several elements, their texts are joined with blank lines. Scripts, styles, noscript, template, svg, iframe, object and <head> content is removed, block elements (paragraphs, headings, list items, table rows…) become line breaks, <br> becomes a newline and whitespace is normalized — so <p>One</p><p>Two</p> yields One\n\nTwo rather than OneTwo.

PDFLoaderCopy Anchor Link

Loads PDF files and extracts text. Optionally splits into one document per page. Requires pdf-parse as a peer dependency.

pnpm add pdf-parse

import { PDFLoader } from '@cogitator-ai/rag';

const loader = new PDFLoader();
const docs = await loader.load('./papers/research.pdf');
console.log(docs[0].metadata);
// { pages: 42 }

const perPage = new PDFLoader({ splitPages: true });
const pageDocs = await perPage.load('./papers/research.pdf');
console.log(pageDocs[0].metadata);
// { pageNumber: 1, totalPages: 42 }

When the PDF declares a title, it is added as metadata.title. In per-page mode, pages without text are skipped and the remaining documents keep their physical pageNumber.

WebLoaderCopy Anchor Link

Fetches a URL and extracts text content. HTML is processed by HTMLLoader (requires cheerio); text/* and JSON responses are kept as raw text; other content types are rejected. Redirects are followed (max 5), responses are decompressed (gzip/deflate/br), decoded with the declared charset and capped at maxResponseBytes. Documents get sourceType: 'web', metadata.url (final URL after redirects), metadata.contentType and, for HTML, metadata.title. Custom headers are merged over the defaults (user-agent: cogitator-rag-webloader and an HTML-first accept).

pnpm add cheerio

import { WebLoader } from '@cogitator-ai/rag';

const loader = new WebLoader({
  selector: 'main',
  headers: { 'User-Agent': 'CogitatorBot/1.0' },
});

const docs = await loader.load('https://example.com/docs/getting-started');

SSRF protectionCopy Anchor Link

By default WebLoader refuses loopback, private (RFC 1918), link-local, CGNAT, multicast and cloud-metadata targets — including IPv4-mapped / NAT64 IPv6 literals, numeric host forms such as http://2130706433/, and hostnames that resolve to such addresses. Resolution is checked at connect time for every redirect hop, so DNS rebinding cannot bypass the check. Set allowPrivateNetwork: true only for trusted URLs.

robots.txtCopy Anchor Link

Pass a robots checker and every URL is checked against its site's robots.txt before it is fetched, redirect targets included. A disallowed URL fails the load with RobotsDisallowedError and is never requested. RobotsPolicy from @cogitator-ai/core reads and caches robots.txt per site by RFC 9309 (longest match wins, * and `--- title: Document Loaders description: Load documents from text files, markdown, JSON, CSV, HTML, PDFs, and web URLs into the RAG pipeline.

OverviewCopy Anchor Link

Document loaders convert raw sources into RAGDocument objects that the pipeline can chunk and embed. Every loader implements the DocumentLoader interface:

interface DocumentLoader {
  load(source: string): Promise<RAGDocument[]>;
  readonly supportedTypes: string[];
}

The source parameter is a file path for file-based loaders, or a URL for WebLoader. Directory paths are supported by TextLoader and MarkdownLoader — they load the matching files directly inside the directory (not subdirectories), in name order. Each document gets a random id, source (absolute path or URL), sourceType and optional metadata, which the pipeline copies onto every chunk.

A pipeline uses a single loader; see Mixing Sources to ingest several file types.

TextLoaderCopy Anchor Link

Loads plain text files (.txt, .text). Accepts a file path or directory.

import { TextLoader } from '@cogitator-ai/rag';

const loader = new TextLoader();

const docs = await loader.load('./data/notes.txt');
const allDocs = await loader.load('./data/text-files/');

No options. Documents get sourceType: 'text'.

MarkdownLoaderCopy Anchor Link

Loads .md and .mdx files. Optionally strips YAML frontmatter and extracts it as metadata.

import { MarkdownLoader } from '@cogitator-ai/rag';

const loader = new MarkdownLoader({ stripFrontmatter: true });
const docs = await loader.load('./docs/');

console.log(docs[0].metadata);
// { title: 'Getting Started', description: 'Quick start guide' }

JSONLoaderCopy Anchor Link

Loads JSON files. Handles both single objects and arrays. Automatically detects content, text, or body fields, or you can specify a custom content field.

import { JSONLoader } from '@cogitator-ai/rag';

const loader = new JSONLoader({
  contentField: 'description',
  metadataFields: ['category', 'author'],
});

const docs = await loader.load('./data/articles.json');

If no content field is found, the entire object is serialized as the document content; non-string field values are serialized as JSON, and array items that are not objects become their string value. source must be a single file.

Auto-detected content fields (in order): content, text, body.

CSVLoaderCopy Anchor Link

Loads CSV files. Each row becomes a separate document. Requires papaparse as a peer dependency.

pnpm add papaparse

import { CSVLoader } from '@cogitator-ai/rag';

const loader = new CSVLoader({
  contentColumn: 'description',
  metadataColumns: ['id', 'category'],
  delimiter: ',',
});

const docs = await loader.load('./data/products.csv');

The first row is the header. If no contentColumn is specified, the first column is used; a contentColumn missing from the header throws. Every document gets metadata.row (1-based data row number) plus the metadataColumns values.

HTMLLoaderCopy Anchor Link

Loads HTML files and extracts text content using CSS selectors. Requires cheerio as a peer dependency.

pnpm add cheerio

import { HTMLLoader } from '@cogitator-ai/rag';

const loader = new HTMLLoader({ selector: 'article' });
const docs = await loader.load('./pages/about.html');

console.log(docs[0].metadata);
// { title: 'About Us' }

The <title> tag is automatically extracted as metadata if present. When the selector matches several elements, their texts are joined with blank lines. Scripts, styles, noscript, template, svg, iframe, object and <head> content is removed, block elements (paragraphs, headings, list items, table rows…) become line breaks, <br> becomes a newline and whitespace is normalized — so <p>One</p><p>Two</p> yields One\n\nTwo rather than OneTwo.

PDFLoaderCopy Anchor Link

Loads PDF files and extracts text. Optionally splits into one document per page. Requires pdf-parse as a peer dependency.

pnpm add pdf-parse

import { PDFLoader } from '@cogitator-ai/rag';

const loader = new PDFLoader();
const docs = await loader.load('./papers/research.pdf');
console.log(docs[0].metadata);
// { pages: 42 }

const perPage = new PDFLoader({ splitPages: true });
const pageDocs = await perPage.load('./papers/research.pdf');
console.log(pageDocs[0].metadata);
// { pageNumber: 1, totalPages: 42 }

When the PDF declares a title, it is added as metadata.title. In per-page mode, pages without text are skipped and the remaining documents keep their physical pageNumber.

WebLoaderCopy Anchor Link

Fetches a URL and extracts text content. HTML is processed by HTMLLoader (requires cheerio); text/* and JSON responses are kept as raw text; other content types are rejected. Redirects are followed (max 5), responses are decompressed (gzip/deflate/br), decoded with the declared charset and capped at maxResponseBytes. Documents get sourceType: 'web', metadata.url (final URL after redirects), metadata.contentType and, for HTML, metadata.title. Custom headers are merged over the defaults (user-agent: cogitator-rag-webloader and an HTML-first accept).

pnpm add cheerio

import { WebLoader } from '@cogitator-ai/rag';

const loader = new WebLoader({
  selector: 'main',
  headers: { 'User-Agent': 'CogitatorBot/1.0' },
});

const docs = await loader.load('https://example.com/docs/getting-started');

SSRF protectionCopy Anchor Link

By default WebLoader refuses loopback, private (RFC 1918), link-local, CGNAT, multicast and cloud-metadata targets — including IPv4-mapped / NAT64 IPv6 literals, numeric host forms such as http://2130706433/, and hostnames that resolve to such addresses. Resolution is checked at connect time for every redirect hop, so DNS rebinding cannot bypass the check. Set allowPrivateNetwork: true only for trusted URLs.

patterns, a 4xx robots.txt allows everything, a 5xx or a network failure allows nothing until the cache expires):

import { RobotsPolicy } from '@cogitator-ai/core';
import { WebLoader } from '@cogitator-ai/rag';

const userAgent = 'NewsBot/1.0 (+https://example.com/bot)';
const loader = new WebLoader({
  headers: { 'user-agent': userAgent },
  robots: new RobotsPolicy({ userAgent }), // matches "User-agent: NewsBot" groups, else "*"
});

Custom LoadersCopy Anchor Link

Implement the DocumentLoader interface to load from any source:

import { randomUUID } from 'node:crypto';
import type { DocumentLoader, RAGDocument } from '@cogitator-ai/rag';

class NotionLoader implements DocumentLoader {
  readonly supportedTypes = ['notion'];

  constructor(private readonly fetchPage: (pageId: string) => Promise<string>) {}

  async load(pageId: string): Promise<RAGDocument[]> {
    const content = await this.fetchPage(pageId);

    return [
      {
        id: randomUUID(),
        content,
        source: `notion://${pageId}`,
        sourceType: 'text',
        metadata: { pageId },
      },
    ];
  }
}

sourceType must be one of 'pdf' | 'html' | 'csv' | 'json' | 'markdown' | 'text' | 'web'.

Pass your custom loader to the pipeline builder:

const pipeline = new RAGPipelineBuilder()
  .withLoader(new NotionLoader(fetchNotionPage))
  // ...
  .build();

await pipeline.ingest('page-id-here');

Mixing SourcesCopy Anchor Link

To ingest several file types through one pipeline, wrap the built-in loaders in a loader that picks one by extension:

import { extname } from 'node:path';
import {
  MarkdownLoader,
  PDFLoader,
  TextLoader,
  WebLoader,
  type DocumentLoader,
  type RAGDocument,
} from '@cogitator-ai/rag';

const byExtension: Record<string, DocumentLoader> = {
  '.md': new MarkdownLoader({ stripFrontmatter: true }),
  '.mdx': new MarkdownLoader({ stripFrontmatter: true }),
  '.pdf': new PDFLoader({ splitPages: true }),
  '.txt': new TextLoader(),
};
const web = new WebLoader();

const mixedLoader: DocumentLoader = {
  supportedTypes: ['md', 'mdx', 'pdf', 'txt', 'http', 'https'],
  async load(source: string): Promise<RAGDocument[]> {
    if (/^https?:\/\//.test(source)) return web.load(source);
    const loader = byExtension[extname(source).toLowerCase()];
    if (!loader) throw new Error(`No loader for ${source}`);
    return loader.load(source);
  },
};