Skip to content

robots.txt blocks ClaudeBot (Anthropic training opt-out)

ClaudeBot is Anthropic's crawler for public web content that could potentially contribute to model training. Anthropic documents it separately from Claude-SearchBot, which supports search, and Claude-User, which handles user-directed retrieval. This check found robots.txt disallowing ClaudeBot.

Why this matters

This is a model-training policy choice, not proof that the site is excluded from Claude search or user-directed retrieval. Anthropic says restricting ClaudeBot signals that future site material should be excluded from its AI model training datasets.

How to fix it

No change is required if the training opt-out is deliberate. If you want to permit ClaudeBot training collection, remove or narrow the ClaudeBot Disallow rule. Review Claude-SearchBot and Claude-User separately for search and user-directed retrieval.

Turn website findings into an action plan

Create an account to start a Full Audit of your website, with supporting evidence and a prioritised action plan.

Start Full Audit