How to block AI crawlers with robots.txt

To block an AI crawler, add a group for its user agent to your robots.txt, such as User-agent: GPTBot followed by Disallow: /. Most AI companies run separate crawlers for training and for search, so you can block training while staying in AI answers: block GPTBot, ClaudeBot and Google-Extended, and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed.

Before you block anything, decide what you want. There are three sensible choices, and each one has a cost.

What are your three choices?

1. Allow every AI crawler. Your pages can be used for training, and assistants can find, quote and link them. For a product that wants to be recommended when someone asks ChatGPT, Claude or Perplexity "what's the best tool for…", this is usually the right call. It's what we do on MakerRadar.

2. Block training, stay in AI search. You opt out of having your content used to train future models, but the assistants' search crawlers can still read your pages and cite them with a link. This is the choice most publishers who care about traffic make.

3. Block every AI crawler. Your content stays out of training and out of AI answers. Google and Bing search are unchanged, because their search crawlers (Googlebot and Bingbot) aren't AI crawlers. The cost is visibility: when someone asks an assistant about your category, it can't read your site, so it relies on what others say about you.

Which crawler is which?

The companies document their crawlers by job. The ones that matter most:

| Company | Training | Search (cites and links) | When a person asks | | --- | --- | --- | --- | | OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | | Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | | Perplexity | — | PerplexityBot | Perplexity-User | | Google | Google-Extended (a control token, not a crawler) | Googlebot (normal search) | — | | Apple | Applebot-Extended (a control token) | Applebot | — | | Common Crawl | CCBot | — | — |

Google-Extended deserves a note. It isn't a separate crawler: Googlebot does the crawling, and the Google-Extended token tells Google whether that content may be used to train Gemini and to ground its answers. Google says it doesn't affect whether or how your site ranks in Google Search (Google's crawler documentation).

For the full list with each company's own description, see the AI crawler list.

What does the robots.txt look like?

Block training, stay in AI search:

``` User-agent: * Allow: /

User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / ```

Several User-agent lines above one Disallow make one group, so every crawler named gets the same rule. Everything not named, including the search crawlers, follows the * group.

Block every AI crawler: name the search and user-request agents too (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and the rest) in the same group.

There are more than twenty AI user agents, and new ones appear every year. Rather than typing them by hand, use our AI crawler robots.txt generator: pick a preset, change any crawler, add private paths and your sitemap, and copy the file. It runs in your browser.

One rule catches people out

A crawler obeys only the most specific group that names it (that's how RFC 9309 defines robots.txt). If your * group disallows /admin and you add a separate group for GPTBot, GPTBot ignores the * group, so your /admin rule no longer applies to it. Repeat any private paths in every named group, or name the allowed crawlers in the * group itself. The generator does this for you.

Can you say what's allowed, not just who?

Yes, with Content Signals, a line Cloudflare proposed in 2025 that states how content may be used after it's fetched:

`` Content-Signal: search=yes, ai-input=yes, ai-train=no ``

search covers search results, ai-input covers using pages to answer a question in real time, and ai-train covers training. It sits inside a user-agent group. It's a statement of your terms, not a technical block; pair it with the Disallow rules above.

Do AI crawlers actually obey robots.txt?

The large companies say their crawlers do, and you can watch them in your server logs. But robots.txt is a request, not a lock: a crawler that ignores it isn't stopped by it. Fetches started by a person (ChatGPT-User, Claude-User, Perplexity-User) may be treated differently, so read each company's documentation. To enforce a block, use your host's or CDN's bot controls. Cloudflare, for example, can block AI crawlers at the edge.

How do you check it worked?

Publish the file at the root of your site: https://yoursite.com/robots.txt.
Test a URL against it, crawler by crawler, with the robots.txt tester.
Run the AI visibility checker to see which AI crawlers can read your homepage, and whether you have an llms.txt.

If you're launching a product, you probably want option 1 or 2: being cited by AI assistants is now one of the ways people find new tools. List your product on MakerRadar and it gets a page written to be read and quoted by them.