Free Tool
AI Crawler Rule Generator
Create clear robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and other AI crawlers in your browser.
GPTBot
OpenAI training crawler
OAI-SearchBot
OpenAI search crawler
ClaudeBot
Anthropic training crawler
Claude-SearchBot
Anthropic search crawler
PerplexityBot
Perplexity search crawler
Google-Extended
Google Gemini training control
Bytespider
ByteDance AI crawler
Your generated rules
# AI crawler policy # Review these rules with your legal and content teams before publishing. # GPTBot: OpenAI training crawler User-agent: GPTBot Disallow: / # OAI-SearchBot: OpenAI search crawler User-agent: OAI-SearchBot Allow: / # ClaudeBot: Anthropic training crawler User-agent: ClaudeBot Disallow: / # Claude-SearchBot: Anthropic search crawler User-agent: Claude-SearchBot Allow: / # PerplexityBot: Perplexity search crawler User-agent: PerplexityBot Allow: / # Google-Extended: Google Gemini training control User-agent: Google-Extended Disallow: / # Bytespider: ByteDance AI crawler User-agent: Bytespider Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
This generates policy text locally. robots.txt is not a security control; use authentication or a firewall for sensitive content.