Robots Meta Tags vs. Robots.txt: Two Different Jobs
Why these two commonly-confused mechanisms solve different problems, and the specific way mixing them up backfires.
Published April 7, 2026
Both have "robots" in the name, both involve telling crawlers what to do, and that's about where the similarity ends. Confusing them is one of the most common technical SEO mistakes, and it's an easy one to make because the failure mode is quiet: nothing throws an error, the site just doesn't get indexed the way you expected.
robots.txt controls crawling
It's a single file at the site root that tells well-behaved crawlers which paths they're allowed to request, before they ever load a page. It operates at the level of whole paths, and it's a guideline crawlers are supposed to respect, not an access control mechanism.
The robots meta tag controls indexing
It's a tag inside one specific page's own <head>, and a crawler only sees it after successfully loading that page. It tells search engines whether this particular page should appear in search results at all, independent of whether other pages can link to it or crawl it.
The mistake that breaks both at once
Blocking a page in robots.txt while also trying to noindex it with a meta tag. These two instructions contradict each other in a way that isn't obvious from either file alone: if robots.txt blocks the page, crawlers never load it, which means they never see the noindex tag telling them not to index it. The disallowed URL can still end up indexed, sometimes with no title or description at all, exactly the outcome the noindex tag was supposed to prevent. For a page you genuinely want out of search results, use noindex alone, and make sure the page stays crawlable so the tag can actually be seen.