Bytespider
ByteDance's data crawler, widely associated with AI training collection. Notorious for aggressive crawl volumes.
| robots.txt token | Bytespider |
| Respects robots.txt | Widely reported ignored |
| Verification | No published IP ranges — requests claiming this UA cannot be verified |
| Our recommendation | Block — Does not reliably take no for an answer — this is tarpit territory, not robots.txt territory. |
Block Bytespider with robots.txt
User-agent: Bytespider
Disallow: /robots.txt is a request, not a wall — requests claiming to be Bytespider from unverified networks should be treated as bad bots. How to verify crawlers by network →
See every Bytespider request hitting your site — live, with per-bot allow / block / tarpit controls. Try TrafficDATA free (100K page views)