What is robots.txt?

Short definition

A file that tells crawlers which paths they may fetch.

Why it matters for Shopify brands

AI visibility requires allowing key AI user-agents on storefront URLs while protecting checkout.

AI visibility work mixes measurement language (share of voice, prompt panels) with technical language (crawlers, schema, Shopping). Clear definitions keep teams from optimizing the wrong surface — for example chasing Google rankings while ChatGPT still cannot fetch product HTML.

Examples in practice

• Shoppers and operators encounter “robots.txt” when researching AI recommendations, ChatGPT Shopping, or technical AI readiness.
• On TopMention pages, “robots.txt” is defined here so guides and ask pages can link to a single owner URL instead of redefining it.

How TopMention uses this term

• Dictionary owns the “What is …?” primary intent for this term.
• When you need a full implementation primer, follow the Learn owner link below.
• If this term names a crawler, the /bots entity page is the deeper owner for User-Agent facts.
• Ask and guide templates link here for definitions instead of competing H1s.

Common mistakes

• Treating the term as interchangeable with classic SEO rankings
• Creating multiple pages that all try to rank for the same definitional query
• Using the jargon in H1s on guide pages (guides should stay category-modified how-tos)

Go deeper

• Read the full implementation guide
• Category guides (applications, not definitions)
• Ask hubs (shopper questions)

Related terms

• GPTBot — OpenAI crawler used to gather web content that may inform ChatGPT and related products.
• AI crawler — A bot operated by an AI company to fetch public web pages.