robots.txt blocks Claude-User (live browsing agent)
Claude-User is the crawler Anthropic's Claude uses when a person asks it to open or read a specific web page in real time, distinct from ClaudeBot (bulk AI training) and Claude-SearchBot (the search index). This check found robots.txt disallowing Claude-User specifically. Blocking Claude-User makes your page invisible at the exact moment someone actively researching your business asks Claude to open it and summarise what it says, arguably the highest-intent moment an AI tool can send you. This gap can exist even if your other Claude-related crawler rules are fine. Your developer needs about 15 minutes. Allow Claude-User in robots.txt unless you deliberately want to prevent Claude from opening pages on a user's request. This crawler is separate from ClaudeBot and Claude-SearchBot, so check your rules for those two as well rather than assuming this one change covers everything.
Why this matters
Blocking Claude-User makes your page invisible at the exact moment someone actively researching your business asks Claude to open it and summarise what it says, arguably the highest-intent moment an AI tool can send you. This gap can exist even if your other Claude-related crawler rules are fine.
How to fix it
Your developer needs about 15 minutes. Allow Claude-User in robots.txt unless you deliberately want to prevent Claude from opening pages on a user's request. This crawler is separate from ClaudeBot and Claude-SearchBot, so check your rules for those two as well rather than assuming this one change covers everything.