robots.txt sets a Crawl-delay that slows indexing on some search engines
If this finding appears in a report, the sampled robots.txt contains a Crawl-delay declaration. Whether it applies depends on the named crawler and matching group. Google does not support Crawl-delay; a declaration alone does not prove slow indexing.
Why this matters
A deliberate delay can protect server availability for crawlers that honour it. An obsolete restriction can reduce their request frequency, but this observation does not measure actual crawl activity or explain lost clicks.
How to fix it
Identify the crawler and applicable robots.txt group, then review server requests, errors and the reason for the restriction before changing it. Preserve intentional limits. Google retired the Search Console crawl-rate control on 8 January 2024; do not look for that setting. For Google, investigate crawl access and server responses. For other crawlers, follow their current documentation. Verify the deployed file and observed requests after any change.
Turn website findings into an action plan
Create an account to start a Full Audit of your website, with supporting evidence and a prioritised action plan.
Start Full Audit