Robots.txt Generator
Create crawl rules and sitemap lines for robots.txt.
What would you like to do next?
Generate a robots.txt file
Robots.txt gives crawlers basic instructions about which parts of a site they may request. Choose an allow-all, block-all or custom path policy, add disallow paths, include a sitemap URL and optionally add common AI crawler blocks. The output is plain text ready to copy into a robots.txt file at the root of your site.
Use rules carefully
Robots.txt controls crawling, not access security. Do not use it to hide private pages, because the file itself is public and crawlers may still know those URLs exist.
Frequently asked questions
Where should robots.txt be placed?
It belongs at the site root, for example https://example.com/robots.txt.
Does Disallow protect private data?
No. It is a crawler instruction, not authentication. Private content should require real access control.
Should I include a sitemap line?
Yes, if you have a sitemap. It helps crawlers discover the canonical sitemap URL quickly.