File Format Guides
Three small text files decide how search engines and AI crawlers see your site. Here's the syntax for each, in plain language.
robots.txt
Lives at yourdomain.com/robots.txt
Tells crawlers which parts of your site they may or may not fetch, per user-agent. It's a request, not an enforced block — well-behaved crawlers (Googlebot, GPTBot) follow it; some don't.
User-agent: * Disallow: /admin/ Allow: / User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xmlCheck your robots.txt →
sitemap.xml
Lives at yourdomain.com/sitemap.xml
A machine-readable list of every URL you want indexed, with optional metadata about when each page last changed. Helps crawlers discover pages faster, especially on large or new sites.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-09-01</lastmod>
</url>
<url>
<loc>https://example.com/tools</loc>
<lastmod>2026-09-05</lastmod>
</url>
</urlset>Check your sitemap →llms.txt
Lives at yourdomain.com/llms.txt
A proposed (not yet official) Markdown file that gives AI crawlers a curated, high-signal summary of your site — what it is, and links to the pages worth reading — since crawling raw HTML at scale is expensive for LLM providers.
# Site Name > One-line description of the site. ## Section - [Page Title](https://example.com/page): One-line summary. - [Another Page](https://example.com/page2): One-line summary.Validate your llms.txt →