Skip to content

robots.txt blocks ChatGPT-User (live browsing agent)

ChatGPT-User is the specific crawler OpenAI uses when a real ChatGPT user asks the assistant to open or read one of your web pages in real time. It's different from GPTBot, which gathers training data in bulk, and OAI-SearchBot, which powers ChatGPT's search results. This check found robots.txt disallowing ChatGPT-User specifically. A site can allow GPTBot and OAI-SearchBot and still be invisible the moment an actual prospect asks ChatGPT to open your page, because that live request uses ChatGPT-User. Even if your other AI crawler settings look fine, this one gap means real-time lookups by a person actively evaluating your business will fail silently. Your developer needs about 15 minutes. In robots.txt, remove the rule disallowing ChatGPT-User, or add an explicit 'User-agent: ChatGPT-User' group with an empty Disallow line, unless you deliberately want to stop ChatGPT from opening pages a user asks it to visit.

Why this matters

A site can allow GPTBot and OAI-SearchBot and still be invisible the moment an actual prospect asks ChatGPT to open your page, because that live request uses ChatGPT-User. Even if your other AI crawler settings look fine, this one gap means real-time lookups by a person actively evaluating your business will fail silently.

How to fix it

Your developer needs about 15 minutes. In robots.txt, remove the rule disallowing ChatGPT-User, or add an explicit 'User-agent: ChatGPT-User' group with an empty Disallow line, unless you deliberately want to stop ChatGPT from opening pages a user asks it to visit.