Robots.txt Generator
Build a valid robots.txt file by choosing which crawlers to allow or block and which paths to disallow.
robots.txt
Ad space
Frequently asked questions
What does robots.txt actually control?+
It tells well-behaved crawlers which paths they may or may not request. It's a guideline, not a security mechanism.
Does Disallow stop a page from appearing in Google?+
Not reliably. A disallowed URL can still be indexed without content if it's linked elsewhere from somewhere else. Use a noindex meta tag instead to keep a page out of search results.
Where do I put this file?+
Save it as robots.txt at your site's root, e.g. https://example.com/robots.txt.