Bots / Diffbot

Diffbot

Commercial web-data crawler used by various AI and knowledge products.

Identity

Name: Diffbot
Operator: Diffbot
User-Agent: Diffbot

Commercial web-data crawler used by various AI and knowledge products. Treat this page as the entity owner for “Diffbot” / User-Agent facts. The Shopify how-to for allowing AI crawlers lives on Learn so we do not compete with that head term here.

Typical robots.txt directive

User-agent: Diffbot
Allow: /

Match the User-Agent string your logs actually see. Some operators use multiple agents (fetch vs training). When in doubt, allow the answer-oriented agents you care about and document intentional denials.

Shopify notes

Optional; prioritize OpenAI/Anthropic/Perplexity agents first for answer visibility.

Implementation steps

Edit robots.txt in Shopify admin (or your SEO app’s robots template)
Allow storefront paths; keep /checkouts, /cart, /account disallowed under User-agent: *
Review security/bot apps for blanket AI blocks
Test on the custom domain customers use (not only *.myshopify.com)
Optional; prioritize OpenAI/Anthropic/Perplexity agents first for answer visibility.

When to allow

You want answer engines operated by this company to fetch public storefront HTML
Your PDPs and collections should be eligible as citation or Shopping sources
robots.txt already allows general crawling but AI agents were accidentally blocked

When teams deny

Your policy is to opt out of training-oriented collection while still allowing answer crawlers
Legal/compliance has mandated a deny for this operator

How to verify

Confirm robots.txt includes a clear rule for Diffbot / Diffbot
Fetch homepage and a hero PDP; expect HTTP 200 with real product HTML
Ensure bot challenges / walls do not trigger for this User-Agent on public paths
Re-run a TopMention tech scan to confirm crawler checks move toward pass