Bots / Anthropic

ClaudeBot

Anthropic crawler that fetches public pages for Claude-related features.

Identity

Name: ClaudeBot
Operator: Anthropic
User-Agent: ClaudeBot

Anthropic crawler that fetches public pages for Claude-related features. Treat this page as the entity owner for “ClaudeBot” / User-Agent facts. The Shopify how-to for allowing AI crawlers lives on Learn so we do not compete with that head term here.

Typical robots.txt directive

User-agent: ClaudeBot
Allow: /

Match the User-Agent string your logs actually see. Some operators use multiple agents (fetch vs training). When in doubt, allow the answer-oriented agents you care about and document intentional denials.

Shopify notes

Allow ClaudeBot on collections and PDPs; keep checkout disallowed.

Implementation steps

Edit robots.txt in Shopify admin (or your SEO app’s robots template)
Allow storefront paths; keep /checkouts, /cart, /account disallowed under User-agent: *
Review security/bot apps for blanket AI blocks
Test on the custom domain customers use (not only *.myshopify.com)
Allow ClaudeBot on collections and PDPs; keep checkout disallowed.

When to allow

You want answer engines operated by this company to fetch public storefront HTML
Your PDPs and collections should be eligible as citation or Shopping sources
robots.txt already allows general crawling but AI agents were accidentally blocked

When teams deny

A security incident requires temporary lockdown (re-open when safe)
You intentionally do not want this operator fetching your storefront

How to verify

Confirm robots.txt includes a clear rule for ClaudeBot / ClaudeBot
Fetch homepage and a hero PDP; expect HTTP 200 with real product HTML
Ensure bot challenges / walls do not trigger for this User-Agent on public paths
Re-run a TopMention tech scan to confirm crawler checks move toward pass