Bots / Common Crawl

CCBot

Common Crawl bot; datasets are widely reused by AI systems.

Identity

Name: CCBot
Operator: Common Crawl
User-Agent: CCBot

Common Crawl bot; datasets are widely reused by AI systems. Treat this page as the entity owner for “CCBot” / User-Agent facts. The Shopify how-to for allowing AI crawlers lives on Learn so we do not compete with that head term here.

Typical robots.txt directive

User-agent: CCBot
Allow: /

Match the User-Agent string your logs actually see. Some operators use multiple agents (fetch vs training). When in doubt, allow the answer-oriented agents you care about and document intentional denials.

Shopify notes

Optional for AI visibility; many stores allow it for open-web inclusion.

Implementation steps

Edit robots.txt in Shopify admin (or your SEO app’s robots template)
Allow storefront paths; keep /checkouts, /cart, /account disallowed under User-agent: *
Review security/bot apps for blanket AI blocks
Test on the custom domain customers use (not only *.myshopify.com)
Optional for AI visibility; many stores allow it for open-web inclusion.

When to allow

You want answer engines operated by this company to fetch public storefront HTML
Your PDPs and collections should be eligible as citation or Shopping sources
robots.txt already allows general crawling but AI agents were accidentally blocked

When teams deny

Your policy is to opt out of training-oriented collection while still allowing answer crawlers
Legal/compliance has mandated a deny for this operator

How to verify

Confirm robots.txt includes a clear rule for CCBot / CCBot
Fetch homepage and a hero PDP; expect HTTP 200 with real product HTML
Ensure bot challenges / walls do not trigger for this User-Agent on public paths
Re-run a TopMention tech scan to confirm crawler checks move toward pass