Robots Meta Tag Generator
Choose index/noindex, follow/nofollow, and other directives to generate a robots meta tag for a single page's <head>.
This controls indexing for one specific page, placed in its own <head>. It's a different job from robots.txt, which controls crawling for whole paths across a site. A page can be crawlable per robots.txt but still excluded from search results via this tag, and the two commonly get confused for each other.
Frequently asked questions
How is this different from robots.txt?+
robots.txt controls crawling for whole paths across a site, at the server level, before a crawler even requests a page. This meta tag controls indexing for one specific page, and the crawler has to actually load the page to see it. A page can be crawlable per robots.txt but still excluded from search results by this tag.
If I want a page excluded from search results, should I use robots.txt or this tag?+
This tag (noindex). Disallowing a page in robots.txt only stops crawling, it doesn't reliably keep an already-linked URL out of search results, sometimes it shows up with no description at all. noindex is the actual mechanism for keeping a specific page out of results.
Can a disallowed page's noindex tag even be seen?+
No, and this is the trap: if robots.txt blocks a page, crawlers never load it and never see the noindex tag telling them not to index it. For noindex to work, the page must be crawlable.