Should you block AI crawlers?
Blocking GPTBot felt obvious in 2023. In 2026, with AI assistants answering buying questions daily, the trade-off looks different: blocking protects your content from training, and it also removes you from answers your customers are reading.
Know the two kinds of AI crawler
Training crawlers (GPTBot, Google-Extended, Applebot-Extended, CCBot) collect content to train future models. Search and answer crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot) fetch content to answer a user’s question right now, often with citation. Blanket-blocking both because you object to the first is how businesses vanish from AI answers.
The case for blocking
If your content IS your product - paid journalism, original research, premium data - training crawlers monetise your work without compensation, and blocking them is a coherent commercial position. Media companies with licensing leverage do exactly this.
The case for allowing
If your website exists to win customers, AI answers are a growing referral and recommendation surface. When an assistant recommends providers in your category, it can only name businesses it can read. For nearly all SMBs, SaaS and service businesses, visibility is worth far more than the content is worth as training data.
A sensible middle: allow answers, choose on training
robots.txt lets you decide per crawler: allow the search/answer set so you appear in live AI results, and make a separate call on the training set. Block your private and app routes from everything regardless. This site allows both sets on public pages and blocks all crawlers from app routes - a deliberate posture, documented in our AI policy.
Whatever you choose, choose it on purpose
The worst position is the accidental one: a block someone added in 2023 that nobody remembers, silently costing recommendations. Audit your robots.txt, know which of the major crawlers can see you, and revisit the policy yearly. AuditHQ’s AI Visibility suite reports exactly which AI crawlers your robots.txt admits and blocks.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
Blocking GPTBot stops OpenAI training on your content. ChatGPT’s live browsing and search use OAI-SearchBot and ChatGPT-User - block those and you disappear from its live answers too. The three are separate lines in robots.txt; the distinction is the whole game.
Do AI crawlers respect robots.txt?
The major named crawlers from OpenAI, Anthropic, Google and Perplexity publicly commit to respecting it, and observed behaviour largely matches. Disreputable scrapers ignore it - but they ignore everything, and robots.txt was never a security control.
Can I block AI training but stay in Google search?
Yes. Google-Extended controls Gemini training only; ordinary Googlebot handles search. Blocking Google-Extended while allowing Googlebot keeps your search rankings and opts out of training - though note AI Overviews inside search are driven by normal search indexing.