Skip to content

robots.txt blocks OAI-SearchBot (ChatGPT live search - separate from GPTBot)

OAI-SearchBot is the crawler that powers ChatGPT Search results, separate from GPTBot which OpenAI uses for training. OpenAI split these two crawlers apart in 2024, and this check found robots.txt disallowing OAI-SearchBot specifically, meaning your content is excluded from live ChatGPT search answers regardless of your GPTBot setting. Most sites that block one of these two crawlers do it by accident, not realising GPTBot and OAI-SearchBot are separate. ChatGPT Search has more than 100 million weekly users, so being structurally excluded from its live results is a real and ongoing distribution loss, not a minor technicality. Your developer needs about 5 minutes. Add an explicit 'User-agent: OAI-SearchBot' group with 'Allow: /' before any broader Disallow rule in robots.txt, or simply remove the rule currently blocking it. Test the fix by fetching robots.txt directly afterwards to confirm the OAI-SearchBot rule now allows access.

Why this matters

Most sites that block one of these two crawlers do it by accident, not realising GPTBot and OAI-SearchBot are separate. ChatGPT Search has more than 100 million weekly users, so being structurally excluded from its live results is a real and ongoing distribution loss, not a minor technicality.

How to fix it

Your developer needs about 5 minutes. Add an explicit 'User-agent: OAI-SearchBot' group with 'Allow: /' before any broader Disallow rule in robots.txt, or simply remove the rule currently blocking it. Test the fix by fetching robots.txt directly afterwards to confirm the OAI-SearchBot rule now allows access.