Scrapy
A purpose-built web-scraping framework. Unlike generic HTTP clients, a Scrapy UA means someone is systematically extracting your content.
| robots.txt token | Scrapy |
| Respects robots.txt | Configurable — obeys robots.txt by default, one setting to disable |
| Verification | None — and serious operators change the UA anyway |
| Our recommendation | Block — A self-declared scraper on a content site is exactly what your bot policy exists for. |
Block Scrapy with robots.txt
User-agent: Scrapy
Disallow: /robots.txt is a request, not a wall — requests claiming to be Scrapy from unverified networks should be treated as bad bots. How to verify crawlers by network →
See every Scrapy request hitting your site — live, with per-bot allow / block / tarpit controls. Try TrafficDATA free (100K page views)