robots.txt generator
Build a valid robots.txt file. Pick a preset, add your rules, then copy or download it.
Runs 100% in your browser robots.txt
How to create a robots.txt file
- Pick a starting point. Choose a preset (allow everything, block everything, or start blank) then adjust.
- Add your rules. List the paths to disallow or allow, and add your sitemap URL.
- Save as robots.txt. Copy or download the file and upload it to the root of your site.
About robots.txt
robots.txt follows the Robots Exclusion Protocol: each block starts with one or more
User-agent lines, followed by Disallow and Allow rules and an
optional Crawl-delay. An empty Disallow: means "allow everything";
Disallow: / blocks the whole site. Paths support * wildcards and
$ end-of-URL anchors in Google and Bing. Always test the result in Google Search
Console's robots.txt report before relying on it for an important site.
Frequently asked questions
- robots.txt is a plain-text file at the root of your site (
example.com/robots.txt) that tells search-engine crawlers which paths they may or may not request. It controls crawling, not indexing — use anoindexmeta tag to keep a page out of results. - At the very root of your domain, reachable at
https://yourdomain.com/robots.txt. It only applies to the host and protocol it is served from. - No. Disallow stops well-behaved crawlers from fetching the page, but a disallowed URL can still appear in results if other pages link to it. To remove a page from the index, allow crawling and add
<meta name="robots" content="noindex">. - Yes — adding a
Sitemap:line helps search engines discover all your URLs. You can include more than one sitemap line.