What it does
meta-externalfetcher is Meta's user action crawler. A live fetch made because a human asked the assistant about your page right now. Closest thing to a real visitor; several of these (Perplexity-User, meta-externalfetcher) ignore robots.txt by design because the request is user-initiated.
User-initiated link fetches for Meta AI; Meta's docs state it may bypass robots.txt because a user requested the fetch. Not confirmed against a live doc fetch this pass — treat as checked.
Full UA: not captured in our July 2026 roster review — match log hits on the token meta-externalfetcher (case-insensitive substring) and verify by IP before acting.
How to verify a hit is really meta-externalfetcher
Verification method (July 2026): Meta ASN 32934 (RDAP/whois); docs page fetch returned 400 this pass
User-agent strings are freely spoofed, so a UA match alone proves nothing. Confirm the source IP belongs to Meta — published IP-range JSON where available, otherwise RDAP/rDNS on the IP. Full method: verify by IP.
Allow or block in robots.txt
Match the exact token meta-externalfetcher in robots.txt:
# Block meta-externalfetcher site-wide
User-agent: meta-externalfetcher
Disallow: /# Explicitly allow meta-externalfetcher
User-agent: meta-externalfetcher
Allow: /For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide.
Vendor documentation: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
Frequently asked questions
What is meta-externalfetcher?
meta-externalfetcher is Meta's user action crawler. A live fetch made because a human asked the assistant about your page right now. Closest thing to a real visitor; several of these (Perplexity-User, meta-externalfetcher) ignore robots.txt by design because the request is user-initiated.
Does meta-externalfetcher respect robots.txt?
No — meta-externalfetcher does not treat robots.txt as binding. Blocking it requires WAF or IP-level rules.
How do I block meta-externalfetcher?
robots.txt cannot stop meta-externalfetcher; block it with WAF or IP-level firewall rules instead.