Twitterbot
Fetches Twitter Card metadata to render link previews when your URL is posted on X.
| robots.txt token | Twitterbot |
| Respects robots.txt | Treats shares as user-triggered fetches |
| Verification | Limited — no published ranges; genuine fetches arrive only when a URL is shared |
| Our recommendation | Allow — Same as every unfurler: blocking it only makes your own links look worse. |
Block Twitterbot with robots.txt
User-agent: Twitterbot
Disallow: /robots.txt is a request, not a wall — requests claiming to be Twitterbot from unverified networks should be treated as bad bots. How to verify crawlers by network →
See every Twitterbot request hitting your site — live, with per-bot allow / block / tarpit controls. Try TrafficDATA free (100K page views)