🤖 AI Crawler Blocker
Generate robots.txt rules to block AI crawlers like GPTBot, ClaudeBot and CCBot from scraping your site.
Pick the AI crawlers you want to keep off your site. The rules below go at the top of your robots.txt file, served at https://yoursite.com/robots.txt.
About this tool
AI crawler blocking uses the same robots.txt file that has controlled search engine crawlers for decades. Each AI company publishes a user-agent string for its crawler, and adding a Disallow: / rule for that agent asks it to stay off your site. This is how you opt out of having your content scraped for model training or AI answer engines.
Select the crawlers you want to block and copy the generated block into the top of your robots.txt. For example, blocking OpenAI's training crawler produces:
User-agent: GPTBot
Disallow: /The list separates training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) from live-fetch agents (ChatGPT-User, Perplexity-User) that only visit when a user clicks a link, so you can decide whether to block scraping while still allowing on-demand access. The "Common training bots" preset selects the crawlers most sites want to block.
FAQ
Does robots.txt actually stop AI crawlers? ›
It is a voluntary standard. Reputable crawlers from OpenAI, Anthropic, Google and others document that they honor it, but robots.txt is not enforced technically, so bad actors can ignore it. For hard blocking you would also need server-side or firewall rules that reject those user-agents.
Will blocking Google-Extended hurt my search ranking? ›
No. Google-Extended only controls whether your content is used to train Gemini and Vertex AI. It is separate from Googlebot, so blocking it does not affect how Google Search crawls or ranks your pages.
What is the difference between a training bot and a user bot? ›
Training bots such as GPTBot and ClaudeBot crawl broadly to collect data for model training. User bots such as ChatGPT-User and Perplexity-User only fetch a page when a person asks the assistant about that specific link. Some sites block training but allow live fetches.
Where do I put these rules? ›
In the plain-text robots.txt file at your domain root, reachable at https://yoursite.com/robots.txt. Add these blocks alongside your existing rules — each User-agent block is independent, so order does not matter.