AI crawler access checklist for candles
Allow GPTBot, ClaudeBot, PerplexityBot and peers so candles Shopify pages can be read for AI answers.
This applies the practice to Candles — for the head how-to, use the Learn link above. Niche context: Candle shoppers ask AI about clean burns, scent strength, soy vs coconut wax, and gifts that feel premium.
Why crawler access decides candles visibility
If AI crawlers cannot fetch your candles pages, models cannot reliably cite or recommend you. Start with the Shopify how-to: Allow GPTBot on Shopify. Security apps and bot challenges are common silent blockers even when robots.txt looks open.
Robots.txt pattern that usually works
Allow GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, and PerplexityBot on storefront paths. Keep checkout, cart, and account disallowed under User-agent: *. See individual crawler pages under /bots for User-Agent strings.
Verify beyond robots.txt
Fetch the homepage and a top candles PDP as an AI user-agent (or via your audit tool). A 200 with real HTML is the bar. Interstitials, JS-only shells with empty HTML, and geo walls all count as failures for citation.
Candles-specific watchouts
Candles brands often add age gates, quiz funnels, or heavy personalization. Ensure the default public HTML still contains product facts without requiring a quiz completion. Collection intros should remain crawlable.
After you change robots
TopMention’s tech audit should flip crawler checks from fail/warn toward pass. Mentions may lag days or weeks; treat access as a prerequisite, not an instant SoV boost.
Policy choices
Some merchants disallow training-oriented agents (for example Google-Extended) while still allowing answer-oriented crawlers. Decide intentionally — do not copy a random disallow list from a forum thread.
Checklist for Candles brands
- Allow GPTBot, OAI-SearchBot, ChatGPT-User
- Allow ClaudeBot / Claude-User and PerplexityBot
- Confirm homepage + PDP HTML fetch without challenges
- Keep /checkouts, /cart, /account disallowed
- Review security app bot rules
- Document any intentional training opt-outs
- Re-test with an AI visibility scan
Niche signals to emphasize
- wax type
- burn time
- fragrance notes
- vessel
- clean burn claims
Definitions
- What is gptbot?
- What is ai crawler?
- What is robots txt?
- What is challenge wall?
- Candles shopper questions
- AI crawler entity pages
Related guides in Candles
- llms.txt checklist for Candles Shopify brands
- ChatGPT Shopping readiness for candles stores
- Product schema priorities for candles
- Citable content patterns in candles
- Shopper prompts AI uses in candles
- Brand entity consistency for Candles Shopify stores
- Collection page AI copy for Candles
- PDP answer blocks for Candles products
- Weekly AI visibility ops for Candles brands
- Proof and citation assets for Candles brands