anthropic-ai

Anthropic · Training claims robots compliance checked — limited vendor docs

What it does

anthropic-ai is Anthropic's training crawler. Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.

Legacy robots.txt token no longer listed in Anthropic's current bot roster (ClaudeBot superseded it). Kept because it still appears in older logs and block lists; near-zero live traffic expected.

Full UA: not captured in our July 2026 roster review — match log hits on the token anthropic-ai (case-insensitive substring) and verify by IP before acting.

How to verify a hit is really anthropic-ai

Verification method (July 2026): none documented (legacy token)

User-agent strings are freely spoofed, so a UA match alone proves nothing. Confirm the source IP belongs to Anthropic — published IP-range JSON where available, otherwise RDAP/rDNS on the IP. Full method: verify by IP.

Allow or block in robots.txt

Match the exact token anthropic-ai in robots.txt:

# Block anthropic-ai site-wide
User-agent: anthropic-ai
Disallow: /
# Explicitly allow anthropic-ai
User-agent: anthropic-ai
Allow: /
Compliance is claimed, not independently confirmed. Anthropic states this bot respects robots.txt; treat the directive as the first line of defense and watch your logs after deploying it.

For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide.

Vendor documentation: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Frequently asked questions

What is anthropic-ai?

anthropic-ai is Anthropic's training crawler. Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.

Does anthropic-ai respect robots.txt?

Anthropic claims anthropic-ai respects robots.txt, but compliance is not independently confirmed.

How do I block anthropic-ai?

Add 'User-agent: anthropic-ai' followed by 'Disallow: /' to your robots.txt. Use the exact token — substring variants of other tokens will not match.

Other bots

GPTBotOAI-SearchBotChatGPT-UserClaudeBotClaude-SearchBotClaude-UserGoogle-ExtendedGoogleOtherGoogle-CloudVertexBotPerplexityBotPerplexity-UserAmazonbotAmzn-SearchBotAmzn-UserApplebotApplebot-ExtendedBytespiderCCBotDuckAssistBotmeta-externalagentmeta-externalfetchercohere-aiAI2BotMistralAI-UserMistralAI-IndexDiffbotPetalBot

AnswerFootprint crawler analytics — bot roster compiled July 2026 from vendor crawler docs and published IP lists (21/28 vendor-doc verified). The analyzer runs 100% client-side — your logs never leave this browser. User-agent strings can be spoofed; treat UA matches as a first pass and verify by IP before acting. Errors in the bot data: report them.