What it does
Applebot-Extended never appears in access logs. It has no HTTP user-agent string — it exists only as a robots.txt User-agent: token. The actual crawling happens under Apple's regular crawler user agents; this token controls whether the content those crawlers already fetch may be used for AI training.
Like Google-Extended: a robots.txt token controlling use of Applebot-crawled data for Apple foundation-model training. Does not fetch pages itself — do not expect it in logs.
Full UA: none exists — Applebot-Extended is a robots.txt-only token with no user-agent string; any log line carrying it is a spoofer.
How to verify a hit is really Applebot-Extended
Verification method (July 2026): n/a — robots.txt control token only
User-agent strings are freely spoofed, so a UA match alone proves nothing. Confirm the source IP belongs to Apple — published IP-range JSON where available, otherwise RDAP/rDNS on the IP. Full method: verify by IP.
Allow or block in robots.txt
Add the token to robots.txt — this is the only place Applebot-Extended does anything:
# Block Applebot-Extended site-wide
User-agent: Applebot-Extended
Disallow: /# Permit AI use of crawled content
User-agent: Applebot-Extended
Allow: /For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide.
Vendor documentation: https://support.apple.com/en-us/119829
Frequently asked questions
What is Applebot-Extended?
Applebot-Extended is a robots.txt control token from Apple — it has no user agent and never appears in access logs; it controls AI-training use of content crawled by Apple's regular crawlers.
Does Applebot-Extended respect robots.txt?
Yes — Apple documents robots.txt compliance for Applebot-Extended (vendor docs reviewed July 2026).
How do I block Applebot-Extended?
Add 'User-agent: Applebot-Extended' followed by 'Disallow: /' to your robots.txt. Use the exact token — substring variants of other tokens will not match.