AR
GEO · 2026-09-08 · 11 min read

The llms.txt experiment — measuring AI crawler behaviour

llms.txt is an emerging convention, not a universally enforced standard. Treat it as an experiment: publish a concise endpoint, log requests, compare it with normal HTML, and report observations by date, user-agent, status code, and response size.

Scope: This is an implementation reference. Validate claims against current Google, vendor, and platform documentation before making a release or policy decision.

What the file can and cannot do

A root `/llms.txt` can provide a machine-readable map of high-value pages. It does not grant access, override robots.txt, guarantee model training, or force an answer engine to cite a page. The useful unit is a stable, low-noise index of canonical resources.

# public/llms.txt
# Keep this file short and human-readable.
# Last reviewed: 2026-09-08

# Abdallah Mekky technical reference
> Free utilities and technical references for SEO, GEO, AEO, and web performance.

## Core references
- [Core Web Vitals](/knowledge/core-web-vitals): field metrics and debugging.
- [JavaScript SEO](/knowledge/seo-audit-checklist): crawl and rendering checks.

## Tools
- [PageSpeed Checker](/tools/pagespeed-checker): Lighthouse metrics and diagnostics.
- [Schema Generator](/tools/schema-generator): JSON-LD templates.

Run an empirical crawler test

Do not claim “GPTBot parses Markdown better” from a single request. Collect access logs for a fixed window, deploy one change, and compare request frequency, response bytes, status distribution, and follow-up page requests. User-agent strings are signals, not proof of model ingestion.

  1. Publish identical resource links in `/llms.txt`, a lightweight HTML index, and the XML sitemap.
  2. Log request timestamp, normalized user-agent, path, status, bytes, referer, and response time.
  3. Use a control period before publishing the file and a treatment period after it.
  4. Compare each crawler separately. Do not pool GPTBot, ClaudeBot, and Perplexity into one “AI” number.
# Nginx log format with enough fields for the experiment
log_format crawler '$time_iso8601 $remote_addr "$request" $status $body_bytes_sent '
                   '"$http_referer" "$http_user_agent" $request_time';
access_log /var/log/nginx/crawler.log crawler;

Lightweight Markdown endpoint

Generate the file from a reviewed allow-list rather than dumping a sitemap. Keep links canonical, descriptions factual, and the response cacheable. In Next.js, a public static file is usually the least fragile implementation.

// app/llms.txt/route.ts
export const dynamic = 'force-static'

export function GET() {
  return new Response("# Technical SEO Reference\n\n> Short, maintained index of canonical resources.\n\n## Tools\n- [PageSpeed Checker](https://www.abdallahmekky.com/tools/pagespeed-checker)\n", {
    headers: {
      'content-type': 'text/plain; charset=utf-8',
      'cache-control': 'public, max-age=3600',
    },
  })
}

Report your own logs with a clear sample size and date range. Until independent replication exists, the defensible conclusion is about your server’s observed requests, not what a model “understands”.