Why UA matching isn't enough
User-agent strings are client-controlled text — spoofing one takes a single curl flag. Common Crawl itself warns that spoofers impersonate CCBot. Scrapers routinely borrow reputable bot names to slip past naive allow-lists, and some bots (Bytespider is widely reported to) rotate user agents entirely. The analyzer is a UA-string first pass: fast and useful for triage, never proof.
The verification method
- Match the source IP against the vendor's published range. Most major AI vendors publish their crawler IPs as JSON (table below). A genuine hit falls inside a published CIDR — we've RDAP-confirmed a real GPTBot hit inside OpenAI's published 74.7.241.0/25 on a production site we operate before trusting it wasn't a spoofer.
- No published list? RDAP/rDNS the IP.
rdap.org/ip/<addr>(orwhois) shows who owns the block — a "Googlebot" from a residential ISP is a spoofer. Reverse DNS helps where vendors document it: Google resolves to*.googlebot.com, Applebot to*.applebot.apple.com— then forward-confirm the hostname resolves back to the same IP. - Treat unverifiable bots accordingly. Some crawlers (Bytespider, cohere-ai, Diffbot) publish no IP ranges at all. You can't positively verify them — decide policy on the assumption that any such UA may be anyone.
Published IP lists (17 bots)
Three gotchas that burn people
1. UA lists rot — new tokens don't substring-match old entries. Vendors ship new-generation crawlers under hyphenated tokens that legacy patterns miss: "Claude-SearchBot" is NOT matched by a "ClaudeBot" pattern, "OAI-SearchBot" is NOT matched by "GPTBot". A stale pattern list let an unthrottled crawler run a 3,737-page enumeration against a production site we operate before anyone noticed. Sweep your patterns whenever traffic looks anomalous.
2. Robots-only tokens in logs = spoofer. Google-Extended and Applebot-Extended have no user agent — they exist only in robots.txt. Any log line "matching" them is fake by definition.
3. Bot inflation skews your human metrics. Cross-check the distribution before trusting visitor counts: on one endpoint we audited, 99.0% of 13,389 hits were bots, and a CDN uniques metric ran ~933× inflated versus real human sessions. If a "traffic spike" is one parameterized endpoint getting hammered, it's breadth-first bot enumeration, not popularity.