Free Tool

robots.txt Generator

Pick allow or block for each crawler, add your sitemap, copy the result. Nothing is sent anywhere — generated live in your browser.

Search Engines

All crawlers (default)

Googlebot (Search)

Bingbot

AI Crawlers

GPTBot (OpenAI training)

ChatGPT-User (live fetch)

OAI-SearchBot (ChatGPT search)

ClaudeBot (Anthropic training)

Claude-Web (live fetch)

PerplexityBot

Google-Extended (Gemini/AI)

CCBot (Common Crawl)

Bytespider (ByteDance)

Applebot-Extended (Apple AI)

Meta external agent

Sitemap URL (optional)Crawl delay in seconds, applies to "All crawlers" (optional)

robots.txt

User-agent: *
Allow: /

Save this as robots.txt and upload it to your site's root directory.

Already have a robots.txt live?

Check exactly which crawlers your current file allows or blocks.