Should You Block AI Crawlers?
No, you should not block AI crawlers if you run a store. OpenAI’s docs say a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers, and that is where shoppers now ask what to buy. Blocking makes sense for publishers who sell attention. It almost never makes sense for a merchant who sells products.
The Case for Blocking AI Crawlers Belongs to Publishers
The case for blocking AI crawlers is a publisher’s case, and for publishers it holds up. Cloudflare measured how much each AI company crawls against how many visitors it sends back. In July 2025, Anthropic crawled about 38,000 pages for every visitor it referred, and OpenAI crawled about 1,100.
Google’s ratio in that same data was 5 to 1. So a news site that lets Googlebot in gets a reader for every few pages crawled, and one that lets ClaudeBot in gets close to nothing back.
That math is why newsrooms shut the door. Press Gazette found 79% of almost 100 top news sites in the US and UK block at least one AI training bot, and 71% block the bots used for live search answers.
Their logic is easy to follow. A publisher’s product is the article itself. When an AI answer sums up a recipe or a news story, the reader never visits and the ad never loads, which is the same squeeze I describe in whether AI search kills organic traffic.
Most of the advice you read about blocking comes from that world. It gets written by publishers, for publishers, and then a store owner copies the block list.
A Store Loses Nothing When an AI Crawler Reads It
A store loses nothing when AI crawlers read its product pages, since an AI answer cannot replace a heated jacket or a mattress base. It can only describe one and point somewhere. You want the pointing to land on you.
That pointing is worth more than it sounds. When an assistant names your product, you get shelf placement you did not pay for, in front of a buyer who asked to be sold to. There is no bid and no coupon involved.
It is also the whole difference between the two business models. A publisher gets paid when someone reads the page. You get paid when someone buys the product, and the more places that describe your product well, the more buyers you get.
Most of the web already acts this way. Ahrefs checked about 140 million websites in 2025 and found 5.89% blocked GPTBot, the most blocked AI bot there is. Blocking is common in news and rare everywhere else.
The buyers AI sends are also good ones. Adobe’s retail data shows AI visitors now convert better than other traffic, and I cover it in my post on whether AI citations convert into customers.
Blocking the Search Bot Pulls Your Store Off the Shelf
Blocking the search bot pulls your store out of ChatGPT search answers. OpenAI’s crawler documentation says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links.”
Picture what that looks like on a Tuesday night. A shopper asks ChatGPT which heated jacket to buy for a Chicago winter. The search bot already indexed your competitor’s product page, and yours was blocked, so the answer names three jackets and none of them are yours. You can’t rank on ChatGPT from behind a closed door.
Not every AI crawler does that job, though, and lumping them together is where most blocking mistakes start:
| Bot | Its Job | What Blocking It Does |
|---|---|---|
| GPTBot, ClaudeBot | Collect training data | Future models learn less about your brand |
| OAI-SearchBot, Claude-SearchBot, PerplexityBot | Build the index AI answers cite | Your pages drop out of those answers |
| ChatGPT-User, Claude-User | Fetch a page when a user asks | Often nothing, since OpenAI says robots.txt “may not apply” |
| Google-Extended | Gemini training and grounding | No effect on Google Search |
Plenty of SEOs stop at “allow the search bots, block the training bots,” and I go further. If you sell products, allow every AI crawler, training bots included.
Training data is how a future model learns your brand exists and what it is known for. A model that never read about your store cannot recommend it from memory, and I walk through that selection process in how ChatGPT picks its sources.
Even Publishers Pay a Price for Blocking
Blocking AI crawlers costs even large publishers about 7% of their weekly visits. A working paper from Rutgers and Wharton tracked news sites before and after they blocked AI bots and found that drop within six weeks.
I will concede the limits of that study. It covers big news publishers, not stores, and an earlier draft reported a larger number that the authors later revised down.
But the direction is the part that matters to you. The sites with the best reason to block still lost traffic when they did it, and some of them at least get licensing money in return. A store gets nothing back.
That is why robots.txt is the first file I open in my AI search services work. A block someone added two years ago can undo everything else on the list.
On Shopify, Check the File and Leave It Alone
On Shopify, the default robots.txt blocks no AI crawlers, so the only job is making sure nobody changed it. Shopify’s help page on editing robots.txt says “the default robots.txt file works for most stores” and warns that a bad edit “can result in loss of all traffic.”
Open yourstore.com/robots.txt in a browser and search the page for GPTBot, OAI-SearchBot, and ClaudeBot. If none of them show up, you are fine. If one does, somebody added a robots.txt.liquid template to your theme, and removing that template puts the default back.
Shopify also runs a second door that robots.txt does not control. Your product data goes to ChatGPT and Copilot through Shopify Catalog, and Shopify says blocking crawlers “affects only open-web discoverability.” Your products can still appear there, but your collection pages, guides, and reviews drop out of the answers, so the feed is no reason to block. I cover that setup in getting Shopify products into ChatGPT Shopping.
The one fair reason to block a bot is server load, and Shopify takes that off your plate too. The same help page says bot management “is handled directly for stores on Shopify, and you don’t need to take any action.” So a crawler hammering your store is Shopify’s problem to solve, not a reason to edit your robots.txt.
Google added a switch of its own in 2026. A Search Console setting now lets you opt out of AI Overviews and AI Mode without leaving Google. Stay opted in, since Google says sites that opt out “will not receive traffic or impressions from our generative AI features.”
Want Your Products in the AI Aisle?
Blocking AI crawlers protects publishers who sell attention. You sell products, and AI assistants are becoming the aisle where shoppers ask for recommendations, so your job is the opposite: stay crawlable and get cited.
If you want to know where you stand, book a call and I will read your robots.txt with you and tell you what I would change first. If you want it handled for you, take a look at my AI search services.
