| Token | Vendor | Purpose | Appears in logs? | robots.txt | Verification | Docs |
|---|---|---|---|---|---|---|
| AI2Bot | Allen Institute for AI | Training | yes | honors robots.txt | none published | docs |
| Amazonbot | Amazon | Training | yes | honors robots.txt | published IP list | docs |
| Amzn-SearchBot | Amazon | AI search | yes | honors robots.txt | published IP list | docs |
| Amzn-User | Amazon | User action | yes | claims robots compliance | published IP list | docs |
| anthropic-ai | Anthropic | Training | yes | claims robots compliance | none published | docs |
| Claude-SearchBot | Anthropic | AI search | yes | honors robots.txt | published IP list | docs |
| Claude-User | Anthropic | User action | yes | claims robots compliance | published IP list | docs |
| ClaudeBot | Anthropic | Training | yes | honors robots.txt | published IP list | docs |
| Applebot | Apple | AI search | yes | honors robots.txt | IP list + rDNS | docs |
| Applebot-Extended | Apple | Training | robots.txt token only — never in logs | honors robots.txt | n/a — robots.txt token | docs |
| Bytespider | ByteDance | Training | yes | ignores robots.txt | none published | none |
| cohere-ai | Cohere | Training | yes | ignores robots.txt | none published | none |
| CCBot | Common Crawl | Training | yes | honors robots.txt | IP list + rDNS | docs |
| Diffbot | Diffbot | Training | yes | ignores robots.txt | none published | docs |
| DuckAssistBot | DuckDuckGo | AI search | yes | honors robots.txt | published IP list | docs |
| Google-CloudVertexBot | User action | yes | honors robots.txt | IP list + rDNS | docs | |
| Google-Extended | Training | robots.txt token only — never in logs | honors robots.txt | n/a — robots.txt token | docs | |
| GoogleOther | Training | yes | honors robots.txt | IP list + rDNS | docs | |
| PetalBot | Huawei | AI search | yes | claims robots compliance | rDNS | docs |
| meta-externalagent | Meta | Training | yes | claims robots compliance | published IP list | docs |
| meta-externalfetcher | Meta | User action | yes | ignores robots.txt | ASN / RDAP | docs |
| MistralAI-Index | Mistral AI | AI search | yes | claims robots compliance | published IP list | docs |
| MistralAI-User | Mistral AI | User action | yes | claims robots compliance | published IP list | docs |
| ChatGPT-User | OpenAI | User action | yes | claims robots compliance | published IP list | docs |
| GPTBot | OpenAI | Training | yes | honors robots.txt | published IP list | docs |
| OAI-SearchBot | OpenAI | AI search | yes | honors robots.txt | published IP list | docs |
| Perplexity-User | Perplexity | User action | yes | ignores robots.txt | published IP list | docs |
| PerplexityBot | Perplexity | AI search | yes | claims robots compliance | published IP list | docs |
Why two bots "never appear in logs": Google-Extended and Applebot-Extended have no HTTP user agent. The actual page fetches happen under Googlebot/Applebot UAs; these tokens exist purely as robots.txt
User-agent: lines that control whether already-crawled content may be used for AI training. A log line matching either name is a spoofer.The three purposes
Training — Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.
AI search — An AI search index that can cite and link your pages in answers (ChatGPT search, Perplexity, Claude search, Alexa, Siri). Blocking removes you from those answer surfaces — for most sites these are wanted crawlers.
User action — A live fetch made because a human asked the assistant about your page right now. Closest thing to a real visitor; several of these (Perplexity-User, meta-externalfetcher) ignore robots.txt by design because the request is user-initiated.