Robots.txt Mistakes That Quietly Deindex Pages You Wanted Crawled
The handful of robots.txt errors that block far more than intended, and why Disallow is the wrong tool for keeping a page out of search results.
Published February 15, 2026
robots.txt looks simple enough that people write it by hand and never look at it again, which is exactly how a site ends up accidentally blocking Googlebot from half its pages for months without anyone noticing. The syntax is forgiving in ways that don't fail loudly, so a small mistake doesn't error out, it just silently does the wrong thing.
The mistake that blocks everything
`Disallow: /` under a matching `User-agent` blocks the entire site from that crawler; it's easy to leave this in from a staging environment's robots.txt and forget to remove it after launch. This single line, left over from a development or staging config, is the single most common reason a brand-new site shows zero organic traffic for weeks after going live: it isn't a ranking problem, it's a crawling problem, and the site simply never gets indexed at all.
Disallow doesn't do what people think it does
A common assumption is that disallowing a path in robots.txt keeps it out of Google's search results entirely. It doesn't, reliably. Disallow tells a well-behaved crawler not to request the page's content, but if the URL is linked from somewhere else, Google can still index the URL itself, sometimes with a "No information is available for this page" result, or a title pulled from the linking page's anchor text. That's usually worse than what someone was trying to prevent.
The actual tool for keeping a specific page out of search results is a `noindex` meta tag or header on that page. This requires the page to be crawlable, which means it must not be disallowed, since a crawler that's blocked from requesting the page can't see the noindex tag telling it not to index it. Blocking a page in robots.txt while also trying to noindex it is a contradiction that usually loses in Google's favor toward keeping it indexed.
Other quiet failure modes
Blocking CSS and JavaScript paths that the page needs to render correctly. Google renders pages to evaluate them, and a disallowed script or stylesheet directory can make a page look broken or incomplete to the crawler even though it looks fine to a person.
Case sensitivity mismatches. Paths in robots.txt are case-sensitive; `/Admin/` and `/admin/` are different rules, and a typo in casing means the intended block silently doesn't apply.
Forgetting the sitemap reference. It's not required, but including `Sitemap: https://example.com/sitemap.xml` in robots.txt gives crawlers an easy, standard place to find it without depending on it being submitted manually everywhere.
- Check robots.txt every time a site moves from staging to production, not just once
- Use noindex, not Disallow, when the goal is keeping a specific page out of search results
- Never disallow the paths that serve a page's own CSS or JavaScript
- Confirm case matches exactly between the rule and the real path