# Vision Tools

URL: https://cogitator.app/docs/browser/vision

> Screenshot capture, accessibility tree traversal, and natural language element interaction.

## Overview

The vision module provides 4 tools for visual interaction with web pages. Two tools capture screenshots as base64-encoded images, and two tools use the accessibility tree and Playwright's built-in selectors to find and click elements by natural language description.

```typescript
import { browserTools } from '@cogitator-ai/browser';

const tools = browserTools(session, { modules: ['vision'] });
```

Or create them yourself. `createVisionTools(session)` returns a typed tuple, so destructuring keeps each tool's parameter and result types:

```typescript
import { createVisionTools, createScreenshotTool } from '@cogitator-ai/browser';

const [screenshot, screenshotElement, findByDescription, clickByDescription] =
  createVisionTools(session);
const single = createScreenshotTool(session);
```

Tools created this way do not start the session on demand like `browserTools()` does — call `session.start()` first.

## Screenshots

### `browser_screenshot`

Captures the current viewport, the full scrollable page, or the first element matching `selector`. By default produces PNG; setting `quality` switches to JPEG. `width` and `height` are read from the image itself.

```typescript
// agent calls:
{ "fullPage": true }
// returns: { image: "base64...", mimeType: "image/png", width: 1280, height: 2400 }

{ "selector": "#hero", "quality": 80 }
// returns: { image: "base64...", mimeType: "image/jpeg", width: ..., height: ... }
```

Tool results reach the model as JSON text, so the model receives the base64 string, not a picture — and a full-page PNG costs a lot of context. To let a vision-capable model actually look at the page, capture the screenshot yourself and pass it as an image input:

```typescript
const shot = await session.page.screenshot({ type: 'png' });

const result = await cogitator.run(agent, {
  input: 'What does the checkout form ask for?',
  images: [{ data: shot.toString('base64'), mimeType: 'image/png' }],
});
```

### `browser_screenshot_element`

Captures a specific element and returns its bounding box coordinates alongside the base64 image.

```typescript
// agent calls:
{ "selector": ".product-card" }
// returns: { image: "base64...", boundingBox: { x: 100, y: 200, width: 300, height: 400 } }
```

## Accessibility-Based Interaction

These tools let agents interact with elements using natural language instead of CSS selectors. This is useful when the page structure is unknown or when CSS selectors would be too brittle.

### `browser_find_by_description`

Walks the page's accessibility tree (Playwright's ARIA snapshot) and matches nodes case-insensitively: the name contains the description, the description contains the name as a whole word (names of 3+ characters), or the role contains the description. Returns up to 50 matches plus the `total` count.

```typescript
// agent calls:
{ "description": "login" }
// returns: {
//   elements: [
//     { role: "button", name: "Login", description: 'button: "Login"' },
//     { role: "link", name: "Login here", description: 'link: "Login here"' }
//   ],
//   total: 2
// }
```

### `browser_click_by_description`

Finds an element by description and clicks it. Tries multiple Playwright strategies in order:

1. `getByRole('button', { name })` — buttons matching the description
2. `getByRole('link', { name })` — links matching the description
3. `getByText(description)` — any element containing the text
4. `getByLabel(description)` — elements with matching label
5. `getByPlaceholder(description)` — inputs with matching placeholder

The first strategy with any match wins; `index` picks which of its matches to click (default 0). With stealth `humanLikeMouse`, the click follows a human-like mouse path. When nothing matches, or `index` is out of range, it returns `{ clicked: false, element: null }`.

```typescript
// agent calls:
{ "description": "Submit order" }
// returns: { clicked: true, element: { description: "Submit order", index: 0 } }

{ "description": "Add to cart", "index": 2 }
// clicks the third "Add to cart" match
```

## Accessibility Tree Helper

The tree used by `browser_find_by_description` is exported for your own code:

```typescript
import { getAccessibilityTree, type AccessibilityNode } from '@cogitator-ai/browser';

const tree: AccessibilityNode | null = await getAccessibilityTree(session.page);
// { role: 'WebArea', name: '', children: [{ role: 'button', name: 'Login' }, ...] }
```

## Vision Flow Example

A typical browsing flow where the agent finds and clicks elements by what they say rather than by CSS selectors:

```typescript
import { Cogitator, Agent } from '@cogitator-ai/core';
import { BrowserSession, browserTools } from '@cogitator-ai/browser';

const session = new BrowserSession({ headless: true });
await session.start();

const agent = new Agent({
  name: 'visual-browser',
  model: 'openai/gpt-6.1-sol',
  tools: browserTools(session, {
    modules: ['navigation', 'vision', 'interaction'],
  }),
  instructions: `You browse the web. Use browser_find_by_description to discover elements,
then browser_click_by_description to interact with them.`,
});

const cogitator = new Cogitator({
  llm: {
    defaultModel: 'openai/gpt-6.1-sol',
    providers: { openai: { apiKey: process.env.OPENAI_API_KEY! } },
  },
});

const result = await cogitator.run(agent, {
  input: 'Go to github.com and find the trending repositories page',
});

console.log(result.output);
await session.close();
```

## Next Steps

* [Tools Reference](/docs/browser/tools) — full parameter reference for all 33 tools
* [Network](/docs/browser/network) — intercept and monitor network traffic