Robots.txt Generator
Generate a robots.txt file with allow/disallow rules, sitemaps, and crawl-delay. Live preview, copy, deep-link, and ready-made presets (allow all, block all, block AI bots, WordPress, Shopify).
Each group applies to its listed user-agents. Use * to match all bots. Rules are applied in order; the first matching group wins for a given bot.
Absolute URLs, one per line. Search engines will fetch these to discover your pages.
About this tool
robots.txt is a small text file you place at the root of your site (https://example.com/robots.txt) that tells crawlers which parts of the site they may visit. Search engines like Google and Bing read it before they crawl; so do AI training crawlers like GPTBot and CCBot.
This generator builds the file rule by rule. Add a group, pick a user-agent (e.g. Googlebot or * for all), then add Allow or Disallow paths. Example: to block everything under /admin for all bots you'd write User-agent: * then Disallow: /admin. The tool also writes Sitemap: lines and the optional Crawl-delay directive.
Presets cover the common one-click setups: allow all, block all (good for staging), block AI bots (GPTBot, ClaudeBot, CCBot, Google-Extended and friends), plus WordPress and Shopify-friendly defaults.
FAQ
Does robots.txt hide pages from Google? ›
No — it only asks crawlers not to crawl. A page that is linked from elsewhere can still be indexed and appear in search results without being crawled. To actually keep a page out of the index, use a noindex meta tag or HTTP header instead.
Is robots.txt case-sensitive? ›
Yes. The directives User-agent, Allow, Disallow, Sitemap, and Crawl-delay must start with a capital letter on most parsers. Paths are case-sensitive too: /Admin and /admin are different.
Does Google honour Crawl-delay? ›
No — Google ignores Crawl-delay. Bing and Yandex do honour it. The directive is included for those crawlers and for transparency.
How do I block AI training crawlers? ›
Use the "Block AI bots" preset, which adds a group disallowing GPTBot, ChatGPT-User, Google-Extended, CCBot, ClaudeBot, Claude-Web, anthropic-ai, Bytespider, Applebot-Extended, and others from accessing your site.
How do I test whether my robots.txt works as expected? ›
Use the Robots.txt Tester & Analyzer — paste the generated file (or fetch it from your live URL), then use the Test a URL Path feature to check whether a specific crawler like Googlebot is allowed or blocked from any path on your site.