robots.txt Generator for AI Crawlers

Pick the crawlers you want to disallow and copy the result into your site's /robots.txt. Search engines are deliberately not listed — you don't want to block those by accident. Facts per crawler live in the crawler directory.

# Generated with trafficdata.site/tools/robots-txt-generator

# Select crawlers on the left to generate Disallow rules.

User-agent: *
Allow: /

Remember: robots.txt is a request. The polite crawlers above honor it — the ones that don't need enforcement. See who actually complies on your site — free for 100K page views.