Robots.txt Tester & Analyzer
Test a robots.txt file against specific URLs — paste or fetch a robots.txt, check which user-agents are allowed or blocked, validate syntax, and debug crawling rules.
See whether a specific crawler would be allowed or blocked from a path, based on the rules above.
About this tool
A robots.txt file sits at https://yoursite.com/robots.txt and tells crawlers (Googlebot, Bingbot, GPTBot, …) which parts of your site they may visit. A single typo — a missing /, a group with no user-agent, a path in the wrong case — can quietly block your whole site or leak pages you meant to hide.
This checker parses the file group by group and explains each rule in plain language. You can paste the file in, open it from disk, or fetch it straight from a live URL. The Test a URL path box answers the question that matters in practice: “will Googlebot be allowed to crawl /admin/settings?
Example: a file containing User-agent: * + Disallow: /admin will report that /admin, /admin/, and /admin/settings are all blocked for every crawler, while /blog is allowed.
FAQ
How are Allow and Disallow rules combined? ›
Within a user-agent group, the most specific matching rule wins. If /public is disallowed but /public/preview is allowed, the preview path is crawlable. If two rules match equally, Allow takes priority over Disallow in Google’s implementation.
Why does the checker say a rule is ignored by Google? ›
Some patterns Google does not support — for example wildcards in the middle of a path without a trailing $, or the Crawl-delay directive (Google ignores it entirely; Bing and Yandex still honour it). The checker flags these so you know which rules actually do something in Google Search.
Can I check a robots.txt that lives on another site? ›
Yes. Paste the full URL (including https://) into the Fetch box. The file is fetched through a public proxy and parsed locally in your browser. Large or unreachable files will show a fetch error.
Does this tool tell me if my pages will appear in Google? ›
No. robots.txt only controls crawling, not indexing. A disallowed page can still be indexed and shown in search results if it is linked from other pages. To keep a page out of the index use a noindex meta tag or response header instead.
How do I create a robots.txt from scratch? ›
Use the Robots.txt Generator — add user-agent groups, allow/disallow rules, sitemaps, and crawl-delay through a form-based UI with ready-made presets for common scenarios like blocking AI bots or WordPress.