Should you block AI crawlers? GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot
Which AI crawlers control visibility in ChatGPT, Claude and Perplexity, which ones collect training data — and how to set up robots.txt without disappearing from AI answers.
In short: if you want ChatGPT, Claude or Perplexity to recommend your company, don’t block their search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Crawlers that collect data for model training, such as GPTBot or ClaudeBot, can be blocked separately — that’s a different decision. In Google, presence in AI Overviews is controlled by the regular Googlebot, not by Google-Extended.
Many companies block “AI bots” in one go — through a plugin, a hosting setting or a robots.txt rule copied from the internet. The result is often the opposite of what they intended: the site disappears from the AI assistants their customers use to find suppliers. Here’s which crawlers do what, and how to make an informed decision.
Three types of AI crawlers
- Search crawlers build the index an assistant uses to answer questions. Blocking them usually means the site won’t be shown or cited in answers.
- Training crawlers collect content that may be used to train future models. Blocking them doesn’t remove the site from that tool’s search.
- User-initiated fetching happens when someone asks an assistant to open a page. Providers note that such requests may not follow robots.txt rules because a person started them.
The main crawlers and their roles
| Company | Crawler | Purpose | Effect of blocking |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT search | The site isn’t shown in ChatGPT search answers |
| OpenAI | GPTBot | Model training | Content isn’t used for training; search is unaffected |
| OpenAI | ChatGPT-User | User-initiated page visits | According to OpenAI, robots.txt may not apply |
| Anthropic | Claude-SearchBot | Claude search | Lower visibility in answers |
| Anthropic | ClaudeBot | Model training | Content isn’t used for training |
| Anthropic | Claude-User | User-initiated fetching | Lower visibility in answers to user questions |
| Perplexity | PerplexityBot | Perplexity search index | The site drops out of the index Perplexity builds answers from |
| Perplexity | Perplexity-User | User-initiated fetching | According to Perplexity, generally ignores robots.txt |
| Googlebot | Search, including AI Overviews and AI Mode | The site disappears from Google, including AI features | |
| Google-Extended (token) | Use of content in other Google AI systems | Doesn’t affect search results or AI Overviews |
Recommended setup for companies that want to be recommended
If your goal is presence in AI answers, the simplest rule is: let search crawlers in, and decide about training crawlers separately. An example robots.txt that blocks model training only:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xmlYou don’t have to block training — many companies deliberately allow it, hoping future models will know their brand better. It’s a business decision, not a technical one. What matters is not blocking search crawlers by accident. Check also firewalls and CDNs (e.g. one-click “block AI bots” settings), hosting defaults and bot protection or CAPTCHA pages.
You can check any site’s robots.txt with our free AI readiness test, or learn more about our AI search visibility service.
Frequently asked questions
Will blocking GPTBot remove my company from ChatGPT?
No. GPTBot collects training data. Visibility in ChatGPT search depends on OAI-SearchBot — that’s the crawler you shouldn’t block if you want to appear in answers.
Can I opt out of AI Overviews but stay in Google?
AI Overviews are part of Google Search, so they can’t be switched off with a separate crawler. Google lets you limit how content is used with nosnippet and max-snippet directives, but these also affect regular results.
Does llms.txt replace robots.txt?
No. llms.txt is a proposed standard for describing a site to language models, not a mechanism for allowing or blocking access. Crawler access is controlled by robots.txt and server settings.
