Bots / TopMention
TopMentionBot
TopMention’s own crawler for AI-visibility technical audits when a merchant requests a scan.
Identity
- Name: TopMentionBot
- Operator: TopMention
- User-Agent:
TopMentionBot/1.0 (+https://topmention.ai/bot)
TopMention’s own crawler for AI-visibility technical audits when a merchant requests a scan. Treat this page as the entity owner for “TopMentionBot” / User-Agent facts. The Shopify how-to for allowing AI crawlers lives on Learn so we do not compete with that head term here.
Typical robots.txt directive
User-agent: TopMentionBot
Allow: /Match the User-Agent string your logs actually see. Some operators use multiple agents (fetch vs training). When in doubt, allow the answer-oriented agents you care about and document intentional denials.
Shopify notes
Only fetches when you (or an authorized party) run a TopMention scan. See /bot for disclosure.
Implementation steps
- Edit robots.txt in Shopify admin (or your SEO app’s robots template)
- Allow storefront paths; keep /checkouts, /cart, /account disallowed under User-agent: *
- Review security/bot apps for blanket AI blocks
- Test on the custom domain customers use (not only *.myshopify.com)
- Only fetches when you (or an authorized party) run a TopMention scan. See /bot for disclosure.
When to allow
- You (or an authorized partner) run TopMention scans and need technical fetches to succeed
- You want accurate llms.txt / schema / crawler evidence in your audit
When teams deny
- You are not using TopMention — denying has no SEO benefit either way
How to verify
- Confirm robots.txt includes a clear rule for TopMentionBot / TopMentionBot/1.0 (+https://topmention.ai/bot)
- Fetch homepage and a hero PDP; expect HTTP 200 with real product HTML
- Ensure bot challenges / walls do not trigger for this User-Agent on public paths
- Re-run a TopMention tech scan to confirm crawler checks move toward pass