Bots / Google
Google-Extended
Google control for Gemini / grounding-related uses of site content (separate from Googlebot SEO crawling).
Identity
- Name: Google-Extended
- Operator: Google
- User-Agent:
Google-Extended
Google control for Gemini / grounding-related uses of site content (separate from Googlebot SEO crawling). Treat this page as the entity owner for “Google-Extended” / User-Agent facts. The Shopify how-to for allowing AI crawlers lives on Learn so we do not compete with that head term here.
Typical robots.txt directive
User-agent: Google-Extended
Allow: /Match the User-Agent string your logs actually see. Some operators use multiple agents (fetch vs training). When in doubt, allow the answer-oriented agents you care about and document intentional denials.
Shopify notes
Decide allow/deny based on your Gemini training/grounding policy; it does not replace Googlebot for Search.
Implementation steps
- Edit robots.txt in Shopify admin (or your SEO app’s robots template)
- Allow storefront paths; keep /checkouts, /cart, /account disallowed under User-agent: *
- Review security/bot apps for blanket AI blocks
- Test on the custom domain customers use (not only *.myshopify.com)
- Decide allow/deny based on your Gemini training/grounding policy; it does not replace Googlebot for Search.
When to allow
- You want answer engines operated by this company to fetch public storefront HTML
- Your PDPs and collections should be eligible as citation or Shopping sources
- robots.txt already allows general crawling but AI agents were accidentally blocked
When teams deny
- Your policy is to opt out of training-oriented collection while still allowing answer crawlers
- Legal/compliance has mandated a deny for this operator
How to verify
- Confirm robots.txt includes a clear rule for Google-Extended / Google-Extended
- Fetch homepage and a hero PDP; expect HTTP 200 with real product HTML
- Ensure bot challenges / walls do not trigger for this User-Agent on public paths
- Re-run a TopMention tech scan to confirm crawler checks move toward pass