If I block these bots, can they still read my site?

A compliant one will stop. The major AI companies' crawlers do respect robots.txt. But the file is a request, not a barrier, so a crawler that chooses to ignore it can still fetch your public pages. If you need to genuinely block a bot, you do that at your CDN or firewall, which enforces the block at the network level rather than relying on good manners.

Will blocking Google-Extended remove me from Google?

No. Google Search uses Googlebot, and Google-Extended is a separate agent that only governs whether your content is used for Google's generative AI training. Blocking Google-Extended leaves your normal search presence untouched. This is one of the most common misunderstandings, which is why the tool flags it directly.

Should I block AI crawlers at all?

That depends on your goal. If you want to be cited when people ask AI assistants for recommendations, blocking works against you. If you would rather your content was not used for training, blocking the relevant bots is reasonable. Many businesses allow the assistant crawlers and block only the bulk dataset scrapers. The tool explains each so you can decide, and does not assume blocking is right.

Where does the file go and how long until it works?

At your site root, yourdomain.co.uk/robots.txt, as a plain-text file. There is only one robots.txt per site. Changes are not instant, they apply the next time each crawler chooses to check the file, which varies from bot to bot. Publishing it does not retroactively remove anything a crawler already collected.

What if a new AI crawler appears?

Your file will not cover it until you add it. robots.txt only affects the user-agents you name, plus any wildcard rule you set. New bots launch and existing ones get renamed, so revisit the file periodically. A catch-all rule can help, but specific named rules are clearer and less likely to catch a crawler you actually wanted.

Free tool

Choose which AI crawlers may read your site

You decide it in robots.txt, a file at yourdomain.co.uk/robots.txt that names each AI crawler and whether it may read your pages. This tool lets you choose crawler by crawler, GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and others, and explains what each one does before you choose. A member of our team is a certified ISO 27001 Lead Auditor and ISO 42001 Auditor, so the tool is candid about what a robots file can and cannot enforce.

Your domain

AI crawlers, one at a time

GPTBot

OpenAI's training crawler, gathers text to train its models.

ChatGPT-User

OpenAI's on-demand fetcher when a ChatGPT user follows a link.

OAI-SearchBot

OpenAI's crawler for its search features and citations.

ClaudeBot

Anthropic's crawler, gathers text to train and power Claude.

Claude-User

Anthropic's on-demand fetcher when a Claude user follows a link.

Google-Extended

Governs use of your content for Google's generative AI. Separate from Google Search.

PerplexityBot

Perplexity's crawler for its answer engine.

CCBot

Common Crawl's crawler. Its public dataset feeds many other AI models.

Applebot-Extended

Governs use of your content for Apple's generative AI.

Bytespider

ByteDance's crawler, associated with TikTok and its AI products.

Default for unlisted crawlers

Sitemap URL (optional)

Publish at yourdomain.co.uk/robots.txt

# Managed AI crawler access for yourdomain.co.uk

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Bytespider
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.co.uk/sitemap.xml

robots.txt is a polite request, not a lock. Compliant crawlers honour it; a determined scraper can ignore it and read your pages anyway. Blocking Google-Extended does not remove you from Google Search or AI Overviews, which run on Googlebot. To genuinely stop a bot you need a CDN or firewall rule, and nothing takes effect until you publish this at your site root.

Runs in your browser. Nothing you type is sent to us or stored.

robots.txt is a polite request, not a lock

Be clear-eyed about what this file can do. robots.txt is voluntary. It is a sign on the door, not a locked door. Well-behaved crawlers from the major AI companies do honour it, but a determined or badly behaved scraper can ignore it completely and simply read your pages anyway, and some do. Publishing these rules does not physically prevent anyone from taking your content. Two more honest caveats. First, as covered above, blocking Google-Extended does not remove you from Google Search or guarantee you vanish from AI Overviews, because those run on Googlebot. Second, this tool only writes the file, it does not upload it, and nothing takes effect until you place it at your site root and the crawlers next visit, which can take time. If you need to actually stop a crawler rather than ask it to leave, that is a job for a CDN or web application firewall rule that blocks the bot at the network level regardless of what robots.txt says, which is outside what this generator does. And bot names change: a crawler renamed or newly launched after you generate the file will not be covered until you update it.

What robots.txt is asking of a crawler

robots.txt is a public notice at the root of your site that names automated visitors, called user-agents, and tells each one which paths it may or may not fetch. For AI specifically, different companies run different bots: OpenAI's GPTBot, Anthropic's ClaudeBot, Google's Google-Extended, Perplexity's PerplexityBot, Common Crawl's CCBot and more. This tool lets you set an Allow or Disallow line for each by name, so you are not making one blanket decision for every AI at once.

The crawlers worth naming, and why they differ

They do not all do the same job. GPTBot and ClaudeBot mainly gather text to train and power AI assistants. CCBot feeds Common Crawl, a public dataset that many other AI models are then built from, so blocking it has wide reach. Google-Extended is a special case: it governs whether Google uses your content for its AI products, and it is separate from normal Google search crawling. The tool explains each bot in plain terms next to its toggle so you choose on the facts, not the name.

Google-Extended does not control your search listing

This trips people up constantly, so it is worth stating plainly. Blocking Google-Extended does not remove you from Google Search and does not, by itself, pull you out of Google's AI answers, because those are driven by the ordinary Googlebot crawler, not Google-Extended. Google-Extended only governs use of your content for training certain generative models. If you want to stay in search while opting out of AI training, that is exactly what naming Google-Extended separately lets you do.

Allowing on purpose is a valid choice

Blocking is not automatically the right move. If you want your business to be quoted and cited when people ask an AI assistant for a supplier, letting the relevant crawlers in is how you become part of that answer. Many businesses will sensibly allow the assistant bots and block only the bulk dataset scrapers, or allow everything. The tool does not push you toward blocking. It lays out each crawler and lets you decide what serves your visibility.

Check what yours says now

The AI crawler checker reads which crawlers your site currently allows

Questions

If I block these bots, can they still read my site?: A compliant one will stop. The major AI companies' crawlers do respect robots.txt. But the file is a request, not a barrier, so a crawler that chooses to ignore it can still fetch your public pages. If you need to genuinely block a bot, you do that at your CDN or firewall, which enforces the block at the network level rather than relying on good manners.
Will blocking Google-Extended remove me from Google?: No. Google Search uses Googlebot, and Google-Extended is a separate agent that only governs whether your content is used for Google's generative AI training. Blocking Google-Extended leaves your normal search presence untouched. This is one of the most common misunderstandings, which is why the tool flags it directly.
Should I block AI crawlers at all?: That depends on your goal. If you want to be cited when people ask AI assistants for recommendations, blocking works against you. If you would rather your content was not used for training, blocking the relevant bots is reasonable. Many businesses allow the assistant crawlers and block only the bulk dataset scrapers. The tool explains each so you can decide, and does not assume blocking is right.
Where does the file go and how long until it works?: At your site root, yourdomain.co.uk/robots.txt, as a plain-text file. There is only one robots.txt per site. Changes are not instant, they apply the next time each crawler chooses to check the file, which varies from bot to bot. Publishing it does not retroactively remove anything a crawler already collected.
What if a new AI crawler appears?: Your file will not cover it until you add it. robots.txt only affects the user-agents you name, plus any wildcard rule you set. New bots launch and existing ones get renamed, so revisit the file periodically. A catch-all rule can help, but specific named rules are clearer and less likely to catch a crawler you actually wanted.