robots.txt Generator
Write the rules that tell search engines which parts of your site to crawl, and add your sitemap line. Upload the file as robots.txt at your site root.
How to use robots.txt Generator
- Choose the crawler, or leave * for all crawlers.
- Enter the paths to allow and block.
- Download robots.txt and upload it to your site root.
Does robots.txt hide pages from search?
It asks crawlers not to visit those paths. Use a noindex tag to keep a page out of search results.
Where should robots.txt go?
At the root of your site, such as https://example.com/robots.txt.