Let Claude search reach your public pages

Choose the bot before changing the rule

Anthropic describes ClaudeBot as a model-development crawler, Claude-SearchBot as supporting search quality and indexing, and Claude-User as fetching pages for user requests. A training opt-out and a search-access decision can therefore be separate. Write down which public content you want discoverable and which uses you intend to permit before editing the file.

Inspect the current policy on the right hostname

Read the live robots.txt and locate all groups relevant to the named bot. Check whether a specific group disallows the whole site or just an intentional private area. Do not assume that changing the generic group changes an existing specific group. Google documents how it selects and combines groups for its crawlers; apply Anthropic policy using its own published guidance and verify the deployed result.

1. Save the current file and identify the source that publishes it.
2. Record the exact bot name and the public URL you are trying to make accessible.
3. Separate a deliberate training restriction from an accidental search restriction.
4. Keep a list of private or operational paths that must stay restricted.

A limited search-access example

This illustrative group allows public paths except the named private area. Adapt the path list to your site and review other groups before use. robots.txt is a crawl preference, not authentication: protect private content with actual access controls. Keep a separate ClaudeBot policy if you intend to restrict training.

User-agent: Claude-SearchBot
Allow: /
Disallow: /private/

Check the CDN as well as robots.txt

An allowed robots rule is not useful if the real page responds with a challenge or an error. Inspect the delivered public content and the relevant CDN or origin logs. A request with a copied user-agent string can test a response path, but that string alone does not prove a request came from Anthropic. Preserve your normal security protections while investigating crawler access.

1. Verify the live robots file after deployment.
2. Check that the public page is available without a login or interactive challenge.
3. Review observed crawler requests and response codes, keeping claimed user agents distinct from verified identity.
4. Retain the before-and-after policy and access observations in the handover.

Measure visibility separately from permission

Permission is the first result to verify. Search visibility is a later result that needs observation. Test a small, repeatable set of relevant questions and record the response date, source links and whether your page appears. Do not report a changed robots rule as new referrals or new customers. AuditHQ can flag public access signals; a scan cannot force an assistant to recommend the business.

Frequently asked questions

See where your website stands

Start with a public-signal screening report in about 60 seconds. No signup or card. Quick Scan samples the website; it does not test every topic in this guide.