Bytespider

ByteDance · Training ignores robots.txt checked — limited vendor docs

What it does

Bytespider is ByteDance's training crawler. Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.

TikTok/Doubao LLM training crawler. No official documentation; widely reported to ignore robots.txt and rotate UAs. Blocking requires WAF rules. Present in our production block lists on both operator properties.

Full UA: not captured in our July 2026 roster review — match log hits on the token Bytespider (case-insensitive substring) and verify by IP before acting.

How to verify a hit is really Bytespider

Verification method (July 2026): none documented (no official docs, no published IP ranges)

No published verification path. ByteDance publishes no IP list, rDNS pattern, or crawler-source documentation for Bytespider as of our July 2026 review — a UA hit cannot be positively confirmed as ByteDance infrastructure. Treat every Bytespider log line as unverified: UA strings are freely spoofed, and for this bot there is nothing to check them against. General triage still applies — see verify by IP.

Allow or block in robots.txt

Match the exact token Bytespider in robots.txt:

# Block Bytespider site-wide
User-agent: Bytespider
Disallow: /
# Explicitly allow Bytespider
User-agent: Bytespider
Allow: /
robots.txt will not stop Bytespider. Per the vendor's own documentation or observed behavior, this bot does not treat robots.txt as binding. If you need it stopped, use WAF or IP-level rules — see verifying and blocking by IP.

For the allow-search-block-training combined pattern (and the Allow-directive gotcha that silently kills carve-outs), see the robots.txt guide. robots.txt governs whether Bytespider may fetch; llms.txt is the separate, curated map AI assistants read once allowed in.

No official vendor documentation is published for this bot.

The ByteDance family

Bytespider is the only ByteDance token in our 34-bot roster — one robots.txt line covers everything ByteDance operates here.

Frequently asked questions

What is Bytespider?

Bytespider is ByteDance's training crawler. Content is copied into datasets used to train foundation models. Blocking costs you nothing in traffic today; allowing is a donation of your content to model weights with no citation or referral in return.

Does Bytespider respect robots.txt?

No — Bytespider does not treat robots.txt as binding. Blocking it requires WAF or IP-level rules.

How do I block Bytespider?

robots.txt cannot stop Bytespider; block it with WAF or IP-level firewall rules instead.

How do I verify Bytespider traffic by IP?

You can't with confidence — ByteDance publishes no IP list or rDNS pattern for Bytespider as of our July 2026 review, so UA-only hits stay unverified.

Related bots

Same vendor, then other training crawlers — the full directory profiles all 34.

CCBotmeta-externalagentcohere-aiAI2BotDiffbotMistralAI-Trainingmeta-externaladsGPTBot

AnswerFootprint crawler analytics — bot roster compiled July 2026 from vendor crawler docs and published IP lists (25/34 vendor-doc verified). The analyzer runs 100% client-side — your logs never leave this browser. User-agent strings can be spoofed; treat UA matches as a first pass and verify by IP before acting. Errors in the bot data: report them.