Skip to content

robots.txt blocks all search engines from crawling the site

robots.txt is a file that tells search engine crawlers which parts of your site they are and are not allowed to visit. This check found a "Disallow: /" rule in this site's robots.txt targeting standard crawlers, User-agent: * and/or specifically Googlebot/Bingbot, which forbids them from crawling any page on the site at all. A site-wide crawl block deindexes the entire site from Google and Bing, making it the most catastrophic self-inflicted SEO mistake possible. It commonly happens after moving a site from a staging or testing environment to production without updating robots.txt, and organic traffic disappears entirely until it is fixed. Your developer should remove or correct the "Disallow: /" rule in robots.txt immediately so search engines are allowed to crawl the site again. This is a one-line fix, but treat it as urgent given the severity, and re-request indexing in Google Search Console once corrected.

Why this matters

A site-wide crawl block deindexes the entire site from Google and Bing, making it the most catastrophic self-inflicted SEO mistake possible. It commonly happens after moving a site from a staging or testing environment to production without updating robots.txt, and organic traffic disappears entirely until it is fixed.

How to fix it

Your developer should remove or correct the "Disallow: /" rule in robots.txt immediately so search engines are allowed to crawl the site again. This is a one-line fix, but treat it as urgent given the severity, and re-request indexing in Google Search Console once corrected.