Should you block AI crawlers?

Know the two kinds of AI crawler

AI crawlers split cleanly into two functional categories, and conflating them is where most robots.txt mistakes come from. Training crawlers - GPTBot, Google-Extended, Applebot-Extended, CCBot - visit your site to collect content that gets folded into a future model's training run, a process that happens on the company's own timeline and has no direct connection to what any individual user is asking about at that moment. Search and answer crawlers - OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot - do something entirely different: they fetch a specific page right now, in direct response to a specific question a real person just typed, often to quote or cite it in the answer that follows. A block written for the first group, out of a reasonable objection to having your content used as training material, frequently sweeps up the second group as well if it is written broadly enough. That is how a business ends up silently absent from AI answers it would otherwise want to appear in, without ever having intended to opt out of anything.

The case for blocking

If your content is the product itself - paid journalism, original research, proprietary data, analysis people pay a subscription to read - then training crawlers are taking your work and folding it into a competitor's product for free, and blocking them is a coherent, defensible commercial position rather than caution for its own sake. Large media organisations with genuine licensing leverage do exactly this, and some have gone further and negotiated paid licensing arrangements instead of blocking outright, because their content has enough standalone commercial value to negotiate over. The common thread is that blocking makes sense when the content is the business, not just marketing for the business - when someone reading it instead of paying for it is a direct loss, not a missed introduction.

The case for allowing

If your website exists to bring in customers rather than to be sold by the page, the calculation flips. AI answers are becoming a real referral and recommendation surface, sitting alongside search results and word of mouth as a place people go before they buy - and when an assistant is asked to recommend a supplier in your category, it can only name and describe businesses it is actually able to read. Blocking every AI crawler on principle does not protect anything of commercial value here; a service page or product listing was never the asset, the customer relationship it leads to is. For the large majority of SMBs, SaaS products and service businesses, the marketing content on the site has essentially no value as training data on its own, but real value as a way of getting recommended - so allowing crawlers costs almost nothing and buys a growing amount.

A sensible middle: allow answers, choose on training

robots.txt lets you set policy per crawler rather than as an all-or-nothing switch, which is the detail most blanket-block setups miss. A workable default is to allow the search-and-answer set so your public pages can be fetched and cited in live AI results, then make a separate, deliberate call on the training set based on how much your content is worth as standalone training material versus how much it is worth as a way of getting found. Whatever you decide on either front, block every crawler - training and answer alike - from account areas, admin routes and anything behind a login, because none of that is content you want summarised in an answer to a stranger. This site allows both sets on its public pages and blocks everything from app routes, a posture chosen deliberately rather than inherited, and documented on the AI policy page so the reasoning is visible rather than assumed.

What each crawler actually does day to day

It helps to know the mechanics behind the names. GPTBot and ClaudeBot crawl broadly and infrequently, gathering pages for a future training run with no immediate connection to any live conversation - blocking them has no effect on today's ChatGPT or Claude answers, only on what a later model version might have learned from you. OAI-SearchBot and PerplexityBot behave more like a traditional search engine's crawler, indexing pages so they can be retrieved and cited quickly when someone asks a live question. ChatGPT-User is different again - it fires when a person pastes your URL directly into a chat and asks the assistant to look at it, a real-time, user-triggered fetch rather than a background crawl, which is why blocking it can break a feature a customer is actively trying to use on the spot. Bingbot sits underneath both Microsoft Copilot and the web results ChatGPT draws on when it searches live, so it is doing double duty for two different assistants at once.

Whatever you choose, choose it on purpose

The worst position by far is the accidental one: a block someone added in 2023 during the first wave of AI-training anxiety, never revisited, quietly costing recommendations every week since with no error message to flag it. Open your robots.txt today, work out which of the crawlers named above it currently admits and which it blocks, and check that the answer matches what you actually intend rather than what a template or a nervous decision three years ago left behind. This is not a set-and-forget file - the AI landscape has moved substantially since most of these blocks were first written, and it will keep moving, so revisit the policy at least yearly rather than assuming an old decision still fits the current situation. AuditHQ's AI Visibility suite reports exactly which AI crawlers your robots.txt currently admits and blocks, so you are working from what is actually configured rather than what you remember setting.

Frequently asked questions