archive.org_bot
Preserves the web in the Wayback Machine.
| robots.txt token | ia_archiver |
| Respects robots.txt | Largely — policy has evolved over the years |
| Verification | Operated by the Internet Archive non-profit |
| Our recommendation | Allow — Preservation has public value and the archive sends occasional rediscovery traffic. |
Block archive.org_bot with robots.txt
User-agent: ia_archiver
Disallow: /robots.txt is a request, not a wall — requests claiming to be archive.org_bot from unverified networks should be treated as bad bots. How to verify crawlers by network →
See every archive.org_bot request hitting your site — live, with per-bot allow / block / tarpit controls. Try TrafficDATA free (100K page views)