MyShopCarts Docs

Appearance

Controlling search and AI crawlers with robots.txt ​

Your store's robots.txt file tells web crawlers — search engines like Google and Bing, and the newer AI crawlers — which parts of your store they may visit. Getting it right helps the pages you care about get discovered, while keeping private and low-value pages out of search results.

Your store already serves a basic robots.txt, and you're free to customize it. Below is the setup we recommend, and how to apply it.

Where your robots.txt comes from ​

Your store's robots.txt has two parts:

  1. A managed baseline. For every store, we automatically keep a baseline in place that blocks known abusive, aggressive crawlers — the kind that crawl heavily to resell your data or feed third-party tools, giving you nothing back while slowing your store down. This part is looked after for you and can't be switched off.
  2. Your own rules. Everything else is yours to control, from your store's admin — including how search engines and AI crawlers treat your content.

Edit your rules ​

In your store's admin:

  1. Go to Stores → Configuration → Design → Search Engine Robots.
  2. Add or change rules in the Edit custom instruction of robots.txt File box.
  3. Save Config.

Your changes take effect right away at https://your-store/robots.txt.

Search engines and AI crawlers ​

There are three kinds of crawler worth knowing about, and you can treat each differently:

  • Search engines (Google, Bing): you almost always want these fully allowed — it's how customers find your store.
  • AI answer crawlers: these fetch a page when someone asks an AI assistant a question, and cite your store with a link in the answer. Allowing them can send you traffic, much like search does. Your store allows these by default.
  • AI training crawlers: these collect content to train AI models. There's no link back and no direct traffic. Your store blocks these by default — a cautious starting point, since training is hard to undo once it's happened. If you'd rather be included (for example, to build your brand's presence in AI tools), you can allow them — see below.

To use these, paste them into the Search Engine Robots box (Stores → Configuration → Design → Search Engine Robots) and Save Config. Here's what each part does, so you can adjust with confidence:

User-agent: *

# Keep private / transactional pages out of search
Disallow: /checkout/
Disallow: /cart/
Disallow: /customer/
Disallow: /sales/
Disallow: /wishlist/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /downloadable/

# Internal search & product-compare pages (no search value)
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/

# Magento housekeeping paths
Disallow: /index.php/
Disallow: /setup/
Disallow: /update/
Disallow: /var/
Disallow: /report/

# Let search engines load images, styles and scripts (needed to render your pages)
Allow: /media/
Allow: /static/
Allow: /pub/media/
Allow: /pub/static/

# Skip filtered / sorted / session URLs (they only duplicate your category pages)
Disallow: /*?*product_list_dir=
Disallow: /*?*product_list_order=
Disallow: /*?*product_list_limit=
Disallow: /*?*product_list_mode=
Disallow: /*?*price=
Disallow: /*?*cat=
Disallow: /*?*___store=
Disallow: /*?*___from_store=
Disallow: /*?*SID=

# Block AI TRAINING crawlers (remove a block to allow that crawler)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /

# Allow AI ANSWER crawlers (they cite your store with a link)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /

# Point crawlers at your sitemap
Sitemap: https://your-store/media/sitemap.xml

Want to allow AI training? ​

Remove the matching User-agent: … / Disallow: / block. For example, deleting the GPTBot block lets that crawler use your content. You stay in full control, crawler by crawler — allow the ones you want, block the ones you don't.

Want to allow deeper crawling of large categories? ​

The starting rules deliberately keep pagination (the ?p=2, ?p=3 … links on big category pages) crawlable, so search engines can walk through to every product. Leave that as-is unless you have a specific reason to change it.

Things to avoid ​

  • Don't block /media/ or /static/. Your images, styles and scripts live there; blocking them stops search engines from rendering your pages properly, which hurts your ranking. That's why the starting rules explicitly allow them.
  • You don't need to hide your admin. It's already protected by sign-in and is never shown in search results.
  • Don't block your whole catalog. Keep your category and product pages crawlable — that's exactly the content you want found.

Tell search engines about your sitemap ​

The Sitemap: line points crawlers at your product and category list. Make sure you've created one — see Creating an XML sitemap — and that its path is /media/sitemap.xml.