Guides / Supplements

AI crawler access checklist for supplements

Allow GPTBot, ClaudeBot, PerplexityBot and peers so supplements Shopify pages can be read for AI answers.

Full primer (owner): /learn/allow-gptbot-shopify

This applies the practice to Supplements — for the head how-to, use the Learn link above. Niche context: Supplement shoppers ask AI for third-party testing, forms that absorb, and brands that won't upset their stomach.

Why crawler access decides supplements visibility

If AI crawlers cannot fetch your supplements pages, models cannot reliably cite or recommend you. Start with the Shopify how-to: Allow GPTBot on Shopify. Security apps and bot challenges are common silent blockers even when robots.txt looks open.

Robots.txt pattern that usually works

Allow GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, and PerplexityBot on storefront paths. Keep checkout, cart, and account disallowed under User-agent: *. See individual crawler pages under /bots for User-Agent strings.

Verify beyond robots.txt

Fetch the homepage and a top supplements PDP as an AI user-agent (or via your audit tool). A 200 with real HTML is the bar. Interstitials, JS-only shells with empty HTML, and geo walls all count as failures for citation.

Supplements-specific watchouts

Supplements brands often add age gates, quiz funnels, or heavy personalization. Ensure the default public HTML still contains product facts without requiring a quiz completion. Collection intros should remain crawlable.

After you change robots

TopMention’s tech audit should flip crawler checks from fail/warn toward pass. Mentions may lag days or weeks; treat access as a prerequisite, not an instant SoV boost.

Policy choices

Some merchants disallow training-oriented agents (for example Google-Extended) while still allowing answer-oriented crawlers. Decide intentionally — do not copy a random disallow list from a forum thread.

Checklist for Supplements brands

  • Allow GPTBot, OAI-SearchBot, ChatGPT-User
  • Allow ClaudeBot / Claude-User and PerplexityBot
  • Confirm homepage + PDP HTML fetch without challenges
  • Keep /checkouts, /cart, /account disallowed
  • Review security app bot rules
  • Document any intentional training opt-outs
  • Re-test with an AI visibility scan

Niche signals to emphasize

  • third-party testing
  • dosage form
  • allergens
  • heavy metals
  • clinical forms

Definitions

Related guides in Supplements

Scan your Supplements store

TopMention is the Shopify AEO/GEO platform — see whether ChatGPT, Claude, Perplexity and Gemini mention your brand, and who's winning your category.

Scan my Shopify store