# Crawler — Regulatory document search engine

> Every regulatory document, one search away.

Crawler is a regulatory document search engine holding 90M+ documents from 1,000+ global sources, organized into 80+ curated collections. An agentic crawler generates its own queries, scores URLs by priority, and runs concurrent agents to keep the index deep and current.

## Key facts
- **90M+** — documents indexed
- **1,000+** — global sources
- **80+** — curated named collections

## What Crawler does

### Deep regulatory coverage
Named collections span bodies like ATSDR, WHO IRIS, EPA ECOTOX, and OECD HPV, so the right corpus is one filter away.

### Agentic crawling
The crawler writes its own queries and scores URLs by priority instead of waiting on a hand-built list.

### Concurrent and current
Parallel crawl agents keep coverage broad and the index fresh.

## Works closely with
- [Sage](https://toxindex.com/platform/sage.md): Conversational answer engine
- [Lists](https://toxindex.com/platform/lists.md): Regulatory chemical lists

## Frequently asked questions

**What does Crawler search?**

Crawler is a regulatory document search engine holding 90M+ documents from 1,000+ global sources across 80+ curated collections.

**How does Crawler stay current?**

An agentic crawler writes its own queries, scores URLs by priority, and runs concurrent agents, so coverage stays broad and the index stays fresh.

**Which regulatory bodies does Crawler cover?**

Named collections span bodies like ATSDR, WHO IRIS, EPA ECOTOX, and OECD HPV, so the right corpus is one filter away.

---
Canonical page: https://toxindex.com/platform/crawler