What it does
Claude-SearchBot is Anthropic's ai search crawler. An AI search index that can cite and link your pages in answers (ChatGPT search, Perplexity, Claude search, Alexa, Siri). Blocking removes you from those answer surfaces — for most sites these are wanted crawlers.
Search-index crawler. NOT substring-matched by 'ClaudeBot' — the missed hyphenated token cost us a 3,737-appid enumeration on a production site we operate (2026-07-06..09). Match it explicitly.
Full UA: not captured in our July 2026 roster review — match log hits on the token Claude-SearchBot (case-insensitive substring) and verify by IP before acting.
How to verify a hit is really Claude-SearchBot
Verification method (July 2026): Published IP list: https://claude.com/crawling/bots.json (fetched live 2026-07-19)
User-agent strings are freely spoofed, so a UA match alone proves nothing. Confirm the source IP belongs to Anthropic — published IP-range JSON where available, otherwise RDAP/rDNS on the IP. Full method: verify by IP.
Allow or block in robots.txt
Match the exact token Claude-SearchBot in robots.txt:
# Block Claude-SearchBot site-wide
User-agent: Claude-SearchBot
Disallow: /# Explicitly allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide.
Vendor documentation: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
Frequently asked questions
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's ai search crawler. An AI search index that can cite and link your pages in answers (ChatGPT search, Perplexity, Claude search, Alexa, Siri). Blocking removes you from those answer surfaces — for most sites these are wanted crawlers.
Does Claude-SearchBot respect robots.txt?
Yes — Anthropic documents robots.txt compliance for Claude-SearchBot (vendor docs reviewed July 2026).
How do I block Claude-SearchBot?
Add 'User-agent: Claude-SearchBot' followed by 'Disallow: /' to your robots.txt. Use the exact token — substring variants of other tokens will not match.