Robots.txt Generator
Generate a robots.txt file with custom allow/disallow rules, crawl delay, and sitemap reference, instantly in your browser.
How to use the Robots.txt Generator
- Enter the user-agent the rules apply to. Use * for all crawlers.
- List the paths to disallow, such as /admin/, one per line.
- List any paths to explicitly allow inside a disallowed folder.
- Optionally add a crawl delay and your sitemap URL, then click Run and save the result as robots.txt in your site's root.
How the Robots.txt Generator works
robots.txt is a plain-text file at the root of a domain that tells compliant crawlers which URLs they may request. The generator writes a User-agent line followed by your Disallow and Allow rules; when rules conflict, Google applies the most specific (longest) matching path. A Sitemap line helps search engines find your XML sitemap.
robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. To keep a page out of search results, allow crawling and add a noindex robots meta tag instead. Google ignores Crawl-delay, though Bing and some others honor it.
Frequently asked questions
- Is this tool free?
- Yes. Generate a robots.txt file at no cost, with no account required.
- Where do I put robots.txt?
- Save it as robots.txt in the root of your website, for example https://example.com/robots.txt.
- What does crawl delay do?
- It asks compliant crawlers to wait a given number of seconds between requests. Not all search engines honor it - Google, for example, ignores it.