Diffbot
Extracts structured data from web pages into a commercial knowledge graph, resold via API to customers including AI products.
| robots.txt token | Diffbot |
| Respects robots.txt | Mixed — customer-requested API fetches may not honor it |
| Verification | Limited — fetches often run on general cloud infrastructure |
| Our recommendation | Your call — Your content becomes structured rows in someone else's commercial product — decide whether that exchange works for you. |
Block Diffbot with robots.txt
User-agent: Diffbot
Disallow: /robots.txt is a request, not a wall — requests claiming to be Diffbot from unverified networks should be treated as bad bots. How to verify crawlers by network →
See every Diffbot request hitting your site — live, with per-bot allow / block / tarpit controls. Try TrafficDATA free (100K page views)