Skip to content

robots.txt blocks Claude-SearchBot (Claude search index)

Claude-SearchBot is the crawler Anthropic uses to build the index that powers Claude's search feature, separate from ClaudeBot (which gathers AI training data) and Claude-User (which fetches a page live when a person asks Claude to look at it). This check found robots.txt disallowing Claude-SearchBot specifically. Blocking Claude-SearchBot keeps your pages out of the results Claude returns when someone searches through it, even if you allow ClaudeBot and Claude-User elsewhere. Since each of these three Claude crawlers serves a different purpose, a block on just one closes off only that slice of visibility, but it's still a real and avoidable gap. Your developer needs about 15 minutes. Allow Claude-SearchBot in robots.txt unless you specifically intend to keep the site out of Claude's search results. Since this crawler is distinct from ClaudeBot and Claude-User, double-check both of those separately rather than assuming one fix covers all three.

Why this matters

Blocking Claude-SearchBot keeps your pages out of the results Claude returns when someone searches through it, even if you allow ClaudeBot and Claude-User elsewhere. Since each of these three Claude crawlers serves a different purpose, a block on just one closes off only that slice of visibility, but it's still a real and avoidable gap.

How to fix it

Your developer needs about 15 minutes. Allow Claude-SearchBot in robots.txt unless you specifically intend to keep the site out of Claude's search results. Since this crawler is distinct from ClaudeBot and Claude-User, double-check both of those separately rather than assuming one fix covers all three.