python-requests
Python's most popular HTTP library, and the default UA of half the quick scraping scripts ever written.
| robots.txt token | python-requests |
| Respects robots.txt | No — unless the script author adds it |
| Verification | None — trivially changed in one line |
| Our recommendation | Block — A default-UA scraper is automation that didn't even try to hide; treat the lazy ones as the floor, not the threat model. |
Block python-requests with robots.txt
User-agent: python-requests
Disallow: /robots.txt is a request, not a wall — requests claiming to be python-requests from unverified networks should be treated as bad bots. How to verify crawlers by network →
See every python-requests request hitting your site — live, with per-bot allow / block / tarpit controls. Try TrafficDATA free (100K page views)