What it does
Claude-User is Anthropic's user action crawler. A live fetch made because a human asked the assistant about your page right now. Closest thing to a real visitor; several of these (Perplexity-User, meta-externalfetcher) ignore robots.txt by design because the request is user-initiated.
Fetches pages when an individual Claude user asks a question.
Full UA: not captured in our July 2026 roster review — match log hits on the token Claude-User (case-insensitive substring) and verify by IP before acting.
How to verify a hit is really Claude-User
Verification method (July 2026): Published IP list: https://claude.com/crawling/bots.json (fetched live 2026-08-22)
Strongest available check: the vendor publishes the crawler's egress IPs. Fetch the list, then confirm the hit's source IP falls inside it — a UA match from any other IP is a spoofer.
User-agent strings are freely spoofed, so a UA match alone proves nothing. Full step-by-step method: verify by IP.
Allow or block in robots.txt
Match the exact token Claude-User in robots.txt:
# Block Claude-User site-wide
User-agent: Claude-User
Disallow: /# Explicitly allow Claude-User
User-agent: Claude-User
Allow: /For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide. robots.txt governs whether Claude-User may fetch; llms.txt is the separate, curated map AI assistants read once allowed in.
Vendor documentation: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
The Anthropic family
Anthropic operates 4 distinct tokens in this roster, and they do not block each other: a robots.txt line for Claude-User does nothing to its siblings — each needs its own User-agent: entry (the robots.txt guide has combined patterns).
| Token | Role | robots.txt |
|---|---|---|
| ClaudeBot | Training | honors robots.txt |
| Claude-SearchBot | AI search | honors robots.txt |
| anthropic-ai | Training | claims robots compliance |
Frequently asked questions
What is Claude-User?
Claude-User is Anthropic's user action crawler. A live fetch made because a human asked the assistant about your page right now. Closest thing to a real visitor; several of these (Perplexity-User, meta-externalfetcher) ignore robots.txt by design because the request is user-initiated.
Does Claude-User respect robots.txt?
Anthropic claims Claude-User respects robots.txt, but compliance is not independently confirmed.
How do I block Claude-User?
Add 'User-agent: Claude-User' followed by 'Disallow: /' to your robots.txt. Use the exact token — substring variants of other tokens will not match.
How do I verify Claude-User traffic by IP?
Fetch Anthropic's published IP list for Claude-User and confirm the hit's source IP falls inside it — a Claude-User user-agent from any other IP is a spoofer.
Related bots
Same vendor, then other user-action fetchers — the full directory profiles all 34.