What it does
meta-webindexer is Meta's ai search crawler. An AI search index that can cite and link your pages in answers (ChatGPT search, Perplexity, Claude search, Alexa, Siri). Blocking removes you from those answer surfaces — for most sites these are wanted crawlers.
Meta's docs: 'navigates the web to improve Meta AI search result quality for users.' Meta's AI-search index crawler — blocking removes you from Meta AI answer surfaces. Lowercase token like its siblings. New on the vendor roster since our 2026-07-19 snapshot.
Full UA: not captured in our July 2026 roster review — match log hits on the token meta-webindexer (case-insensitive substring) and verify by IP before acting.
How to verify a hit is really meta-webindexer
Verification method (July 2026): Meta ASN 32934 (RDAP/whois); no published IP list. Docs fetched live 2026-08-22
No per-crawler IP list is published; the practical check is network ownership — look up the hit's source IP via RDAP/whois and confirm it announces from the vendor's network (ASN) rather than a residential proxy or cloud range.
User-agent strings are freely spoofed, so a UA match alone proves nothing. Full step-by-step method: verify by IP.
Allow or block in robots.txt
Match the exact token meta-webindexer in robots.txt:
# Block meta-webindexer site-wide
User-agent: meta-webindexer
Disallow: /# Explicitly allow meta-webindexer
User-agent: meta-webindexer
Allow: /For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide. robots.txt governs whether meta-webindexer may fetch; llms.txt is the separate, curated map AI assistants read once allowed in.
Vendor documentation: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
The Meta family
Meta operates 4 distinct tokens in this roster, and they do not block each other: a robots.txt line for meta-webindexer does nothing to its siblings — each needs its own User-agent: entry (the robots.txt guide has combined patterns).
| Token | Role | robots.txt |
|---|---|---|
| meta-externalagent | Training | claims robots compliance |
| meta-externalfetcher | User action | ignores robots.txt |
| meta-externalads | Training | claims robots compliance |
Frequently asked questions
What is meta-webindexer?
meta-webindexer is Meta's ai search crawler. An AI search index that can cite and link your pages in answers (ChatGPT search, Perplexity, Claude search, Alexa, Siri). Blocking removes you from those answer surfaces — for most sites these are wanted crawlers.
Does meta-webindexer respect robots.txt?
Meta claims meta-webindexer respects robots.txt, but compliance is not independently confirmed.
How do I block meta-webindexer?
Add 'User-agent: meta-webindexer' followed by 'Disallow: /' to your robots.txt. Use the exact token — substring variants of other tokens will not match.
How do I verify meta-webindexer traffic by IP?
No per-crawler IP list is published; look up the source IP via RDAP/whois and confirm it announces from Meta's network before trusting the user agent.
Related bots
Same vendor, then other ai search crawlers — the full directory profiles all 34.