Does Diffbot respect robots.txt?

Diffbot fetches are commonly governed with User-agent: Diffbot robots.txt rules. Review Diffbot’s developer documentation for product context and contact paths.

What does Diffbot do with pages?

Diffbot specializes in turning web pages into structured records for knowledge graphs and data products used in AI and research workflows—not a consumer chat app brand.

Does blocking Diffbot affect Google rankings?

No. Diffbot is independent of Google Search crawlers.

Is Diffbot a training crawler like GPTBot?

It is closer to dataset/extraction infrastructure. The business model is structured data and APIs rather than a single end-user assistant.

Should news publishers block Diffbot?

Many do when licensing is strict; others allow public pages. Align with content licensing and business development agreements.

How do I allow Diffbot only on certain paths?

Use Allow and Disallow path rules under User-agent: Diffbot (for example Allow: /blog and Disallow: /account). Test path order carefully.

Diffbot

Diffbot · Extracts structured data from the web for knowledge graphs and AI products

What is Diffbot?

Diffbot operates crawlers and extraction systems that turn web pages into structured records for knowledge graphs, research, and commercial AI data products. The Diffbot user-agent appears in logs when their systems fetch pages. Diffbot provides developer documentation for its APIs and products; robots.txt remains a standard way for sites to express crawl preferences. Diffbot is categorized here under dataset/other because its primary role is structured extraction and data products rather than a single consumer chat brand.

How to block Diffbot

Add this group to robots.txt to disallow Diffbot. Path rules can be narrowed if you only need to protect parts of the site.

User-agent: Diffbot
Disallow: /

How to allow Diffbot

For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.

User-agent: Diffbot
Allow: /

Should you block Diffbot?

If you do not want third-party structured extraction of public pages, Disallow Diffbot. If open data reuse is acceptable, Allow. Diffbot is less about “chat citations” and more about machine-readable extraction—align the decision with your content licensing posture. robots.txt should still be paired with legal terms and technical access controls for non-public content.

Category: Dataset / other

Official documentation (Diffbot)

Last verified: 2026-07-18. Re-check operator docs after major crawler announcements.

Diffbot FAQ

Diffbot fetches are commonly governed with User-agent: Diffbot robots.txt rules. Review Diffbot’s developer documentation for product context and contact paths.