What it does
cohere-ai is Cohere's training crawler. Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.
No official crawler docs, UA spec, or IP list published; robots compliance undocumented (false = no documented commitment, not observed defiance). Token appears in community block lists and in our production robots/Caddy lists.
Full UA: not captured in our July 2026 roster review — match log hits on the token cohere-ai (case-insensitive substring) and verify by IP before acting.
How to verify a hit is really cohere-ai
Verification method (July 2026): none documented
User-agent strings are freely spoofed, so a UA match alone proves nothing. Confirm the source IP belongs to Cohere — published IP-range JSON where available, otherwise RDAP/rDNS on the IP. Full method: verify by IP.
Allow or block in robots.txt
Match the exact token cohere-ai in robots.txt:
# Block cohere-ai site-wide
User-agent: cohere-ai
Disallow: /# Explicitly allow cohere-ai
User-agent: cohere-ai
Allow: /For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide.
No official vendor documentation is published for this bot.
Frequently asked questions
What is cohere-ai?
cohere-ai is Cohere's training crawler. Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.
Does cohere-ai respect robots.txt?
No — cohere-ai does not treat robots.txt as binding. Blocking it requires WAF or IP-level rules.
How do I block cohere-ai?
robots.txt cannot stop cohere-ai; block it with WAF or IP-level firewall rules instead.