SEO / Crawling

robots.txt Generator

Create a robots.txt file with rules for each crawler, sitemap lines, and presets to allow everything, block a staging site, or block AI training crawlers.

robots.txt Generator: robots.txt tells crawlers which paths they may fetch, as RFC 9309 standardises: each group names user agents, then Disallow and Allow rules, where the most specific matching rule wins. The AI training preset adds a group for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, and meta-externalagent, the names their operators document for opting out of training. The generator warns about paths without a leading slash, relative sitemap URLs, and rules that block every crawler. Runs 100% locally in your browser with zero server file uploads.

Category
Web tools
Runs
In your browser
Cost
Free · no sign-up
Availability
Ready to use
robots.txt generatorLocal processing

Runs entirely in your browser

Group 1
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Put the file at the root of your site, such as https://example.com/robots.txt. It asks crawlers to stay out; it does not hide or protect anything, and Google ignores Crawl-delay. The AI crawler names were checked on 2 October 2026; companies add new ones, so check their documentation.

Common mistakes

Blocking CSS and JavaScript stops search engines from rendering pages properly; blocking a page you want removed stops crawlers seeing its noindex tag; leaving a staging site's Disallow: / in place after launch hides the whole site. Check the live file after every deployment.

To test an existing file, use the robots.txt checker; to check the sitemap it points to, the sitemap checker.

Wildcards

Google and Bing support * for any characters and $ for the end of the URL: Disallow: /*.pdf$ blocks every PDF. The longest matching rule wins, so Allow: /shop/sale/ overrides Disallow: /shop/.

How to use it

  1. Start from a preset, or edit the groups of user agents and paths.
  2. Add your sitemap URLs.
  3. Copy or download robots.txt and put it at the root of your site.

Privacy & limitations

The file is made in your browser.

Related tools

Robots.txt CheckerSitemapMeta TagsCSP generator

Frequently asked questions

Does robots.txt block pages from Google search?

It stops crawling, not indexing: a blocked page can still appear in results if other sites link to it. To keep a page out of search, allow crawling and add a noindex meta tag.

Will blocking AI crawlers remove my content from AI tools?

It asks those crawlers not to collect your pages from now on; it does not remove anything already collected, and only crawlers that honour robots.txt will comply.

Does Crawl-delay work?

Bing and some other crawlers respect it; Google ignores it and adjusts its crawl rate automatically.

Free tool · runs in your browser · no account required