robots.txt for AI crawlers (Shopify, 2026)

robots.txt is necessary and insufficient. It is still the first control plane AI operators honor.

Recommended posture for most DTC brands

• Allow answer-oriented AI user-agents on storefront paths
• Disallow checkout, cart, account under User-agent: *
• Decide training-oriented agents (Google-Extended, etc.) intentionally
• Verify with fetches + logs

Entity index: /bots. How-to: Allow GPTBot on Shopify.

Example skeleton

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /checkouts
Disallow: /cart
Disallow: /account

Adjust to your threat model. Pair with security-app allowlists (security apps guide).

Governance

Keep a short internal doc: which agents are allowed, who approved training opt-outs, last verification date.