cURL
The standard command-line HTTP client. In your logs it means a script or a person fetched the page outside a browser — uptime checks, scrapers, and exploit probes all look the same here.
| robots.txt token | curl |
| Respects robots.txt | No — tools don't read robots.txt; the person driving them decides |
| Verification | None — the UA is one `-A` flag away from saying anything |
| Our recommendation | Block — On public content pages there's no browsing reason for it — and the honest curls are the least of your worries; classify it as automation at the edge. |
Block cURL with robots.txt
User-agent: curl
Disallow: /robots.txt is a request, not a wall — requests claiming to be cURL from unverified networks should be treated as bad bots. How to verify crawlers by network →
See every cURL request hitting your site — live, with per-bot allow / block / tarpit controls. Try TrafficDATA free (100K page views)