Should you remove crawl-delay from robots.txt?

Which crawlers use crawl-delay?

Google documents crawl-delay as unsupported. Bing supports the directive, and a robots.txt delay takes precedence over its Crawl Control settings. Anthropic also supports crawl-delay for its bots. Removing the line therefore changes crawler-specific behaviour; it is not a universal way to make Google index a site faster.

Find the rule that actually applies

Open the robots.txt file on the affected hostname, not a copy stored in an old project folder. Record the User-agent group containing the delay, its value, and the neighbouring Allow and Disallow rules. Then compare it with the crawler named in your logs. A rule in a ClaudeBot group is a different decision from one in the generic group. Subdomains may serve different files.

1. Save the current live file and identify who owns its generation: CMS, application, CDN or static hosting.
2. List the important URLs whose freshness matters, such as a service page, new article or product listing.
3. Check recent crawler requests and server errors before assuming the delay is the bottleneck.

Choose a change using server evidence

Keep a deliberate rate limit if removing it would create a real availability problem. If the delay is historical and no longer needed, change only that directive in the relevant group. For Bing, consider its Crawl Control interface instead. For Google, investigate response times, error rates and crawl access rather than looking for a crawl-delay setting. Do not remove private-path restrictions while cleaning up an unrelated delay.

Verify the deployed policy

After publishing, retrieve the live file again and compare it with the saved version. Check the public hostname and any redirecting hostname separately. The useful outcome is a correctly served policy plus healthy crawler requests, not merely a successful Git commit.

1. Confirm the intended delay changed and the required Disallow rules remain.
2. Check that the affected public pages still return their expected content and status.
3. Watch crawler requests, response latency and errors over a comparable period.
4. Use indexing and search-performance evidence to assess freshness separately from crawl activity.

What a crawl-delay change cannot promise

A crawler being permitted or being able to request more pages does not guarantee indexing, ranking or AI citations. If the page is inaccessible, duplicated or unhelpful for the query, removing a delay will not repair those issues. For a client handover, record the original purpose, the exact changed group, the verification date and the signal you will compare next. A public scan can identify the directive; it cannot establish your server capacity from that line alone.

Frequently asked questions

See where your website stands

Start with a public-signal screening report in about 60 seconds. No signup or card. Quick Scan samples the website; it does not test every topic in this guide.