GPTBot, ClaudeBot, PerplexityBot and the other AI crawlers, and how to allow or block each

AI companies crawl the web under named user agents, and most run separate ones for training, for search and for fetches a person asks for. OpenAI has GPTBot (training), OAI-SearchBot (ChatGPT search) and ChatGPT-User (user requests); Anthropic has ClaudeBot, Claude-SearchBot and Claude-User; Perplexity has PerplexityBot and Perplexity-User. You allow or block each one with its own group in robots.txt, so you can stay in AI answers while opting out of training.

Why do AI companies run more than one crawler?

Because they do different jobs, and site owners want different things from each. A crawler that collects training data feeds future models. A search crawler builds the index an assistant uses to find and cite pages today. A user-triggered fetcher grabs a single page because someone asked the assistant to read it.

If you block a training crawler, you opt out of having your content used to train models, but you can still appear in answers. If you block a search crawler, you drop out of that assistant's search results, which most product makers don't want. Knowing which is which is the whole game.

Which AI crawlers should you know, by company?

These are the names you'll see most often, with each company's own description.

OpenAI

From OpenAI's crawler documentation:

• GPTBot collects content that may be used to train OpenAI's models.
• OAI-SearchBot finds pages for ChatGPT's search features. Block it and your pages can't appear in ChatGPT search answers.
• ChatGPT-User fetches a page when a ChatGPT user asks for it. OpenAI says robots.txt rules may not apply, because a person started the request.

Anthropic

From Anthropic's support article on its crawlers:

• ClaudeBot collects public web content that may be used for training.
• Claude-SearchBot indexes content to improve Claude's search results.
• Claude-User fetches pages when a Claude user asks a question that needs them.

Anthropic says all three respect robots.txt, and blocking one doesn't affect the others.

Perplexity

From Perplexity's help center:

• PerplexityBot builds Perplexity's search index and links to sites in its answers. Perplexity says it isn't used to train AI models.
• Perplexity-User fetches pages when a person asks a question. Perplexity says that because a user requested the fetch, it generally ignores robots.txt.

Google

• Google-Extended is not a crawler. It's a robots.txt token: Google's ordinary crawlers fetch the page, and this token controls whether that content may be used to train Gemini models and to ground answers in Gemini apps and Vertex AI. Google says it doesn't affect inclusion or ranking in Google Search.
• AI Overviews and AI Mode are part of Google Search, so they follow Googlebot and Google's snippet controls, not Google-Extended. Blocking Googlebot removes you from Search entirely.

Apple

From Apple's Applebot page:

• Applebot crawls for Apple's search features, such as Siri and Spotlight suggestions.
• Applebot-Extended, like Google-Extended, does not crawl anything. It controls whether content Applebot has already crawled may be used to train Apple's generative models. Disallowing it doesn't remove you from Apple's search features.

Common Crawl and others

• CCBot is Common Crawl's crawler. Common Crawl publishes an open archive of the web that many AI developers use as training data, so blocking CCBot is one way to keep new pages out of that archive.
• Other names you'll see include meta-externalagent and meta-externalfetcher (Meta), Amazonbot (Amazon), Bytespider (ByteDance), MistralAI-User (Mistral), DuckAssistBot (DuckDuckGo), cohere-ai (Cohere) and YouBot (You.com). Check each company's documentation for exactly what its agent is used for, since these descriptions change.

How do you allow, block or split them in robots.txt?

robots.txt works in groups. Each group starts with one or more User-agent: lines, followed by Allow: and Disallow: rules. Under the standard, RFC 9309, a crawler uses the group that names it, or the User-agent: * group if none does. Within that group, the longest matching rule wins, and Allow wins a tie.

One detail catches people out: a crawler that has its own group ignores the * group completely. If you write a group for GPTBot, any rules you put under * no longer apply to it.

To opt out of AI training but stay in AI search, a site might use these lines:

• User-agent: GPTBot
• User-agent: ClaudeBot
• User-agent: Google-Extended
• User-agent: Applebot-Extended
• User-agent: CCBot
• Disallow: /

followed by a separate group that lets the search agents in:

• User-agent: OAI-SearchBot
• User-agent: Claude-SearchBot
• User-agent: PerplexityBot
• Allow: /

Each of those lines sits on its own line in the real file, with a blank line between the two groups. To block a crawler from only part of a site, use a path instead of /, such as Disallow: /drafts/.

What are Content Signals?

Content Signals are a newer, optional line in robots.txt, proposed by Cloudflare in 2025 and published at contentsignals.org. Instead of saying who may crawl, they say what the content may be used for after it's collected. There are three signals:

• search: building a search index and showing links and short excerpts.
• ai-input: using the content as input to an AI answer, for example to ground or cite it.
• ai-train: training or fine-tuning AI models.

A line such as Content-Signal: search=yes, ai-input=yes, ai-train=no states your preferences in one place. They are a statement of terms, not a technical block: a crawler still has to choose to honor them.

Does blocking AI crawlers actually work?

For the major, well-documented crawlers, mostly yes. OpenAI, Anthropic, Google, Apple and Perplexity all say their crawlers follow robots.txt. There are real limits, though:

• User-triggered fetchers are treated differently. OpenAI and Perplexity both say their user-request agents may not follow robots.txt, on the reasoning that a person, not a crawler, asked for the page.
• robots.txt is a request, not a lock. A crawler that ignores it, or doesn't identify itself, isn't stopped by it. In August 2025 Cloudflare reported seeing undeclared crawlers it attributed to Perplexity on sites that had blocked PerplexityBot; Perplexity disputed the report. If you need real enforcement, use your CDN's bot controls or a firewall.
• Blocking isn't retroactive. Content already collected for training isn't removed when you add a rule.
• Blocking has a cost. Every search agent you block is an assistant that can no longer cite or recommend you.

For most products, the useful default is to let the search agents in. Whether to allow training is a choice about your content, and it's yours to make. MakerRadar, for its part, welcomes AI crawlers for search and training alike and explains what its own bot fetches on its crawler page. If showing up in AI answers is your goal, our guide on how to show up in ChatGPT answers covers what comes after access.

Test your robots.txt

These free tools need no sign-up:

• The robots.txt tester reads a site's robots.txt and, for the exact URL you give it, shows whether Googlebot, Bingbot and each of the AI crawlers it knows (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more) may fetch it, with the rule that decides. It also reports Sitemap lines, Content Signals and Crawl-delay, and lines crawlers will skip, such as typos and rules before any User-agent.
• The AI visibility checker looks at the bigger picture: robots.txt rules for 20+ AI crawlers, the search crawlers AI answers rely on, Content Signals, llms.txt and a Markdown version of the page, snippet rules that stop answer engines quoting you, and readable text without JavaScript.

Launching a product? List it on MakerRadar.