Reference
AI Crawler Directory
Every known AI crawler user-agent, what it's for, and whether it respects robots.txt. Updated as new AI crawlers appear — last reviewed September 2026.
| User-Agent | Company | Purpose | Type |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content to train future OpenAI models. | Trains AI models |
| ChatGPT-User | OpenAI | Fetches a page live when a ChatGPT user shares a link or browses. | Live fetch / citation |
| OAI-SearchBot | OpenAI | Crawls and indexes pages for ChatGPT search / SearchGPT results. | AI search index |
| ClaudeBot | Anthropic | Crawls content to train Claude models. | Trains AI models |
| Claude-Web | Anthropic | Fetches a page live when Claude browses or cites a source. | Live fetch / citation |
| anthropic-ai | Anthropic | Legacy Anthropic crawler user-agent, still seen on some hosts. | Trains AI models |
| PerplexityBot | Perplexity | Crawls and indexes pages that Perplexity answers can cite. | AI search index |
| Perplexity-User | Perplexity | Fetches a page live when a Perplexity user asks about it directly. | Live fetch / citation |
| Google-Extended | Controls use of your content for Gemini and Google AI features — separate from classic Googlebot indexing. | Trains AI models | |
| Applebot-Extended | Apple | Controls use of your content for Apple Intelligence features. | Trains AI models |
| Bingbot | Microsoft | Classic Bing search crawler — also feeds Bing Copilot / Copilot Search. | AI search index |
| CCBot | Common Crawl | Builds the open Common Crawl dataset, which many LLMs (including early GPT models) trained on. | Trains AI models |
| Bytespider | ByteDance | Crawls content to train ByteDance/TikTok AI models. | Trains AI models |
| Amazonbot | Amazon | Crawls content partly to support Alexa and Amazon AI answers. | AI search index |
| FacebookBot | Meta | Crawls content that may be used to train Meta AI models. | Trains AI models |
| meta-externalagent | Meta | Fetches links shared on Meta platforms and supports Meta AI features. | Live fetch / citation |
Example
How to allow or block a crawler
Add a dedicated User-agent block to your robots.txt for each crawler you want to control — it doesn't inherit from your general User-agent: * rules.
# Block AI training crawlers, keep AI search/citation bots User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / # Allow crawlers that cite you in AI answers User-agent: PerplexityBot Allow: / User-agent: OAI-SearchBot Allow: /
Not sure which crawlers your current robots.txt allows or blocks? The robots.txt AI Crawler Checker reads your file and shows the answer for every crawler on this page.