Verify by IP — a user agent is not proof

Anyone can put "GPTBot" in a request header. Real verification means confirming the source IP belongs to the vendor. Here's the method, plus every published IP list we track (July 2026).

Why UA matching isn't enough

User-agent strings are client-controlled text — spoofing one takes a single curl flag. Common Crawl itself warns that spoofers impersonate CCBot. Scrapers routinely borrow reputable bot names to slip past naive allow-lists, and some bots (Bytespider is widely reported to) rotate user agents entirely. The analyzer is a UA-string first pass: fast and useful for triage, never proof.

The verification method

  1. Match the source IP against the vendor's published range. Most major AI vendors publish their crawler IPs as JSON (table below). A genuine hit falls inside a published CIDR — we've RDAP-confirmed a real GPTBot hit inside OpenAI's published 74.7.241.0/25 on a production site we operate before trusting it wasn't a spoofer.
  2. No published list? RDAP/rDNS the IP. rdap.org/ip/<addr> (or whois) shows who owns the block — a "Googlebot" from a residential ISP is a spoofer. Reverse DNS helps where vendors document it: Google resolves to *.googlebot.com, Applebot to *.applebot.apple.com — then forward-confirm the hostname resolves back to the same IP.
  3. Treat unverifiable bots accordingly. Some crawlers (Bytespider, cohere-ai, Diffbot) publish no IP ranges at all. You can't positively verify them — decide policy on the assumption that any such UA may be anyone.

Published IP lists (20 bots)

BotVendorIP list / ranges
GPTBotOpenAIhttps://openai.com/gptbot.json
OAI-SearchBotOpenAIhttps://openai.com/searchbot.json
ChatGPT-UserOpenAIhttps://openai.com/chatgpt-user.json
ClaudeBotAnthropichttps://claude.com/crawling/bots.json
Claude-SearchBotAnthropichttps://claude.com/crawling/bots.json
Claude-UserAnthropichttps://claude.com/crawling/bots.json
GoogleOtherGooglehttps://developers.google.com/static/crawling/ipranges/common-crawlers.json
PerplexityBotPerplexityhttps://www.perplexity.com/perplexitybot.json
Perplexity-UserPerplexityhttps://www.perplexity.com/perplexity-user.json
AmazonbotAmazonhttps://developer.amazon.com/amazonbot/ip-addresses
Amzn-SearchBotAmazonhttps://developer.amazon.com/amazonbot/searchbot-ip-addresses/
Amzn-UserAmazonhttps://developer.amazon.com/amazonbot/live-ip-addresses/
ApplebotApplehttps://search.developer.apple.com/applebot.json
CCBotCommon Crawlhttps://index.commoncrawl.org/ccbot.json
DuckAssistBotDuckDuckGohttps://duckduckgo.com/duckassistbot.json
MistralAI-UserMistral AIhttps://mistral.ai/mistralai-user-ips.json
MistralAI-IndexMistral AIhttps://mistral.ai/mistralai-index-ips.json
OAI-AdsBotOpenAIhttps://openai.com/adsbot.json
Google-AgentGooglehttps://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json
Google-GeminiNotebookGooglehttps://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json

Field note: what a UA-spoofing scanner actually looks like

Observed on this site, 2026-08-15 (Cloudflare edge logs): the site took 621 client-error requests in a day, and the eight most common user-agent strings on those errors were all AI-crawler names — Amazonbot, Amzn-SearchBot, GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended — each of them appearing on the same credential-probe paths, i.e. one scanner rotating borrowed names. The tell wasn't the UA, it was the paths: /.env, /.aws/credentials, /wp-admin/install.php, /.gcloud/credentials, /actuator/configprops — credential probes no real crawler fetches. Every request got a 404/405. Two lessons that generalize: (1) a UA-only crawler report would have shown a surge of "AI crawler interest" that was one hostile scanner, and (2) the Google-Extended hits were spoofed by definition (gotcha #2 below) — the fastest single filter you have.

Three gotchas that burn people

1. UA lists rot — new tokens don't substring-match old entries. Vendors ship new-generation crawlers under hyphenated tokens that legacy patterns miss: "Claude-SearchBot" is NOT matched by a "ClaudeBot" pattern, "OAI-SearchBot" is NOT matched by "GPTBot". A stale pattern list let an unthrottled crawler run a 3,737-page enumeration against a production site we operate before anyone noticed. Sweep your patterns whenever traffic looks anomalous.
2. Robots-only tokens in logs = spoofer. Google-Extended and Applebot-Extended have no user agent — they exist only in robots.txt. Any log line "matching" them is fake by definition.
3. Bot inflation skews your human metrics. Cross-check the distribution before trusting visitor counts: on one endpoint we audited, 99.0% of 13,389 hits were bots, and a CDN uniques metric ran ~933× inflated versus real human sessions. If a "traffic spike" is one parameterized endpoint getting hammered, it's breadth-first bot enumeration, not popularity.

AnswerFootprint crawler analytics — bot roster compiled July 2026 from vendor crawler docs and published IP lists (25/34 vendor-doc verified). The analyzer runs 100% client-side — your logs never leave this browser. User-agent strings can be spoofed; treat UA matches as a first pass and verify by IP before acting. Errors in the bot data: report them.