robots.txt for AI crawlers — allow search, block training

The pattern most sites want: stay citable in AI search answers (a referral channel) while opting out of model-training scrapes (a donation). Exact tokens below, compiled July 2026 from vendor docs.

Why per-purpose, not per-vendor

Most AI vendors run multiple crawlers with different jobs: OpenAI's GPTBot trains models while OAI-SearchBot powers ChatGPT search citations; Anthropic's ClaudeBot trains while Claude-SearchBot indexes for search. Blocking "everything from OpenAI" throws away citation traffic to stop a training crawl you could have blocked alone. Decide per purpose.

Block the training crawlers

Every robots-respecting training-purpose token in our roster (10 tokens — including the two robots-only control tokens Google-Extended and Applebot-Extended, which is precisely where they DO work):

# --- Block AI model-training crawlers ---
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: AI2Bot
Disallow: /

Keep AI-search crawlers in

If your robots.txt has broad Disallow rules, name the search crawlers explicitly so they keep citing you:

# --- Allow AI-search / citation crawlers ---
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Applebot
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: MistralAI-Index
Allow: /

User-agent: PetalBot
Allow: /

The Allow-directive gotcha (this silently kills carve-outs)

Never anchor an Allow with $. Under RFC 9309, when Allow and Disallow both match a URL, the longest rule wins — but a $-anchored Allow like Allow: /blog$ only matches the single exact URL, so for everything under /blog/… your shorter Disallow: wins and the carve-out silently dies. Use unanchored prefixes:

# WRONG — carve-out dies for /blog/anything
User-agent: GPTBot
Disallow: /
Allow: /blog$

# RIGHT — unanchored prefix
User-agent: GPTBot
Disallow: /
Allow: /blog

Then verify against the served file (curl https://yoursite/robots.txt) — CDNs cache robots.txt past your deploy.

Exact tokens or nothing

New-generation tokens don't substring-match old ones. "Claude-SearchBot" is not matched by a "ClaudeBot" line; "OAI-SearchBot" is not matched by "GPTBot"; "Amzn-SearchBot" is newer than most published block lists. Copy tokens exactly from the directory — a stale list once let an unthrottled crawler enumerate 3,737 pages on a production site we operate.

What robots.txt can't do

These bots don't treat robots.txt as binding: Perplexity-User, Bytespider, meta-externalfetcher, cohere-ai, Diffbot. User-action fetchers (Perplexity-User, meta-externalfetcher) skip it by design because a human requested the page. Stopping them takes WAF or IP-level rules — verify by IP first, then block at the edge.

Aim directives at the pain, not the whole site

A bot hammering one parameterized endpoint (/search?q=…, /item?id=N) is breadth-first enumeration, not interest — we've watched a single crawler enumerate 2,038 unique parameter values in 24 hours. Disallow the parameter surface, keep the citable content pages open:

User-agent: *
Disallow: /*?

Compliant crawlers pick up robots.txt changes within about a crawl cycle (OpenAI documents ~24h for OAI-SearchBot); give a new directive a day before concluding it's ignored.

AnswerFootprint crawler analytics — bot roster compiled July 2026 from vendor crawler docs and published IP lists (21/28 vendor-doc verified). The analyzer runs 100% client-side — your logs never leave this browser. User-agent strings can be spoofed; treat UA matches as a first pass and verify by IP before acting. Errors in the bot data: report them.