cohere-ai

Cohere · Training ignores robots.txt checked — limited vendor docs

What it does

cohere-ai is Cohere's training crawler. Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.

No official crawler docs, UA spec, or IP list published; robots compliance undocumented (false = no documented commitment, not observed defiance). Token appears in community block lists and in our production robots/Caddy lists.

Full UA: not captured in our July 2026 roster review — match log hits on the token cohere-ai (case-insensitive substring) and verify by IP before acting.

How to verify a hit is really cohere-ai

Verification method (July 2026): none documented

No published verification path. Cohere publishes no IP list, rDNS pattern, or crawler-source documentation for cohere-ai as of our July 2026 review — a UA hit cannot be positively confirmed as Cohere infrastructure. Treat every cohere-ai log line as unverified: UA strings are freely spoofed, and for this bot there is nothing to check them against. General triage still applies — see verify by IP.

Allow or block in robots.txt

Match the exact token cohere-ai in robots.txt:

# Block cohere-ai site-wide
User-agent: cohere-ai
Disallow: /
# Explicitly allow cohere-ai
User-agent: cohere-ai
Allow: /
robots.txt will not stop cohere-ai. Per the vendor's own documentation or observed behavior, this bot does not treat robots.txt as binding. If you need it stopped, use WAF or IP-level rules — see verifying and blocking by IP.

For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide. robots.txt governs whether cohere-ai may fetch; llms.txt is the separate, curated map AI assistants read once allowed in.

No official vendor documentation is published for this bot.

The Cohere family

cohere-ai is the only Cohere token in our 34-bot roster — one robots.txt line covers everything Cohere operates here.

Frequently asked questions

What is cohere-ai?

cohere-ai is Cohere's training crawler. Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.

Does cohere-ai respect robots.txt?

No — cohere-ai does not treat robots.txt as binding. Blocking it requires WAF or IP-level rules.

How do I block cohere-ai?

robots.txt cannot stop cohere-ai; block it with WAF or IP-level firewall rules instead.

How do I verify cohere-ai traffic by IP?

You can't with confidence — Cohere publishes no IP list or rDNS pattern for cohere-ai as of our July 2026 review, so UA-only hits stay unverified.

Related bots

Same vendor, then other training crawlers — the full directory profiles all 34.

AI2BotDiffbotMistralAI-Trainingmeta-externaladsGPTBotClaudeBotanthropic-aiGoogle-Extended

AnswerFootprint crawler analytics — bot roster compiled July 2026 from vendor crawler docs and published IP lists (25/34 vendor-doc verified). The analyzer runs 100% client-side — your logs never leave this browser. User-agent strings can be spoofed; treat UA matches as a first pass and verify by IP before acting. Errors in the bot data: report them.