Free tool
AI crawler robots.txt generator
Choose, crawler by crawler, which AI companies may read your site for search and which for training, and copy the robots.txt lines.
Crawler by crawler
Each AI company runs separate crawlers for training and for answering questions. Change any one; Googlebot and Bingbot are never named, so search indexing is unchanged.
| Crawler | What it's for | Rule |
|---|---|---|
| GPTBotOpenAI | TrainingCollects content that may be used to train OpenAI's models. | |
| OAI-SearchBotOpenAI | AI searchFinds pages to show and link in ChatGPT search; it isn't used for training. | |
| ChatGPT-UserOpenAI | On requestFetches a page when a ChatGPT user or one of its agents asks for it. | |
| ClaudeBotAnthropic | TrainingCollects content that may be used to train Anthropic's Claude models. | |
| Claude-SearchBotAnthropic | AI searchIndexes pages to improve the search results Claude gives its users. | |
| Claude-UserAnthropic | On requestFetches a page when a Claude user asks a question that needs it. | |
| Google-ExtendedGoogle | Opt-out tokenNot a crawler: controls whether pages Google already crawls may train Gemini models and ground Gemini's answers; it doesn't affect Google Search. | |
| GoogleOtherGoogle | TrainingGoogle's generic crawler for one-off and research crawls by its product teams, apart from Google Search. | |
| ApplebotApple | AI searchApple's search crawler for Siri, Spotlight and Safari suggestions; blocking it takes the site out of those. | |
| Applebot-ExtendedApple | Opt-out tokenNot a crawler: controls whether pages Applebot crawls may train Apple's foundation models; it doesn't affect Apple's search features. | |
| PerplexityBotPerplexity | AI searchIndexes pages to show and link in Perplexity's answers; it isn't used for training. | |
| Perplexity-UserPerplexity | On requestFetches a page when a Perplexity user asks a question that needs it. | |
| meta-externalagentMeta | TrainingCrawls the web for Meta's AI, including training its models. | |
| meta-externalfetcherMeta | On requestFetches a page when someone using Meta's AI products asks for it. | |
| AmazonbotAmazon | TrainingCrawls to improve Amazon's products and services, and may train Amazon's AI models. | |
| CCBotCommon Crawl | TrainingBuilds Common Crawl's free web archive, an open corpus many AI models are trained on. | |
| MistralAI-UserMistral AI | On requestFetches a page when a Le Chat user asks for it; it isn't used for training. | |
| DuckAssistBotDuckDuckGo | AI searchFetches pages for DuckDuckGo's AI-assisted answers, which cite their sources; it isn't used for training. | |
| cohere-aiCohere | TrainingA token Cohere's crawling uses; Cohere doesn't publicly document what it collects. | |
| BytespiderByteDance | TrainingByteDance's crawler, widely reported to collect data for its AI models. | |
| YouBotYou.com | AI searchCrawls pages for You.com's AI search and answers. |
One per line, each starting with "/" — kept out of every crawler (at most 20).
Its full address, written at the end of the file.
Your robots.txt
Every AI crawler may read your site. Publish it at the root of your site (/robots.txt), replacing the AI lines you have.
# robots.txt: which AI crawlers may fetch this site, and what for.
# Crawlers named over Disallow: / are kept out of the whole site. Every other
# crawler, search engines included, follows the User-agent: * group.
User-agent: *
Allow: /
How do I block ChatGPT from my site?
AI companies crawl under named user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended …); a robots.txt group per agent allows or blocks each one.
What it checks
Every AI crawler we know, by company, with what it collects for: training, search or a person's request
A preset to allow all, block training only, or block every AI crawler
Paths to keep private from every crawler, and your sitemap
Content Signals: search, AI input and AI training, said in the file itself
Use it from your AI assistant
Claude, ChatGPT, Cursor and any other MCP client can run this check for you. Add the server below, then ask it to check a site: the tool is generate_ai_robots_txt. No sign-up and no key.
https://api.makerradar.io/mcpOr call the REST API, which answers the same report as JSON:
curl -X POST https://api.makerradar.io/v1/tools/ai-robots-txt-generator \
-H "content-type: application/json" -d '{"preset":"block_training","disallowPaths":["/admin"]}'Get a page that passes, free
Every product on MakerRadar gets a permanent page with a clean title and description, structured data, an FAQ, a Markdown version for AI assistants and a place in our llms.txt — and a backlink you can earn.
Related tools
robots.txt tester
Paste any URL on a site and see, for Google, Bing and every major AI crawler, whether its robots.txt lets them fetch it — and which line decides.
Reads the page
AI visibility checker
Find out whether AI assistants and answer engines are allowed to read your site — and can understand it when they do.
Reads the page
llms.txt generator
Paste your site's address: we check its /llms.txt against the spec, or draft one from your homepage that you can save as it is.
Reads the page
More free tools
Website grader
Paste a URL and get one grade for the whole site: found on Google, quoted by AI, shared well and safe to visit, with the fixes that matter most first.
Reads the page
Launch checklist
Everything a product's website needs before launch day, in order, each step with the free tool that checks it, then the steps for launch day itself.
Runs in your browser
Launch directory planner
Pick what you need (free, a dofollow link, a minimum Domain Rating) and get a short, ranked plan of where to launch, a few a week, with each directory's DR.
Answers from MakerRadar's own catalogue
Domain availability checker
Type a name and see which startup domains are free, and for the taken ones, when they were registered and with which registrar.
Looks up the domain, never visits the site
Alternatives finder
Paste a product's address and see the products listed here that do the same job, in the order its alternatives page ranks them.
Answers from MakerRadar's own catalogue
SEO checker
Paste a URL and see what Google reads on that page — and what to fix first.
Reads the page
AI citation checker
Paste a URL and see how easy that page is for ChatGPT, Perplexity and Google's AI Overviews to quote, signal by signal, with what to add first.
Reads the page
What AI sees
Paste a URL and read the page the way an AI assistant does: the HTML your server sends, turned into Markdown, with its outline, and whether your content only appears once JavaScript runs.
Reads the page
Agent readiness checker
Paste your site's address and see which of the files and signals AI agents look for it publishes: the same set this site publishes for them.
Reads the page
Social card preview
Paste a URL and see the card X, LinkedIn, Facebook and Slack will show when someone shares it — and why it might show no image at all.
Reads the page
Logo & favicon grabber
Paste a URL and see every icon the site declares, its real size, and the one a directory or app would pick: shown from the site's own address, never copied.
Reads the page
Structured data checker
Paste a URL and see every schema.org type the page declares in JSON-LD, which rich results it can earn, and the fields each one is missing.
Reads the page
Sitemap checker
Paste your site's address: we find its sitemap the way search engines do, read it, and tell you what would stop them using it.
Reads the page
Broken link checker
Paste a URL and see every link on that page that's broken or redirects: up to 50 links, same-site first, each with its status and anchor text.
Reads the page
Redirect checker
Paste a URL and see every hop it takes before it lands: the status code, the address and what changed, with loops and long chains flagged.
Reads the page
SERP snippet preview
Paste a URL and see how its page would show in Google, with where the title and description would be cut and how to fit them.
Reads the page
Keyword generator
Start from what your product is and get the searches — and the questions people ask AI — that lead to products like it.
Runs in your browser
Domain rating checker
See how strong a domain's backlink profile is — the number launch directories are ranked by.
Looks up the domain, never visits the site
Security headers checker
See which protections a site sends with every page — and which it's missing.
Reads the page
Email DNS checker (SPF, DMARC, DKIM)
Will mail from this domain land in the inbox? Check its SPF, DMARC and DKIM in one go.
Looks up the domain, never visits the site
Domain age checker
How old is a domain, who registered it and when does it expire — straight from the registry.
Looks up the domain, never visits the site
SSL certificate checker
Who issued the site's certificate, when it expires, and whether the zone allows that issuer.
Looks up the domain, never visits the site
Hosting & CDN lookup
Where a site is served from: its platform, its edge network and the origin behind it.
Looks up the domain, never visits the site
Tech stack detector
Find out what a site is built with and which third-party scripts it loads.
Reads the page
Frequently asked questions
How do I block AI crawlers with robots.txt?
Add a group per crawler, for example `User-agent: GPTBot` then `Disallow: /`. Each AI company runs separate agents for training and for search, so you can stay in AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) while opting out of training (GPTBot, ClaudeBot, Google-Extended).
Do AI crawlers obey robots.txt?
The major companies say their crawlers do, but robots.txt is a request, not a lock. Agents that fetch a page because a person asked may be treated differently by their operators; check each company's documentation.
Should I block AI crawlers?
Blocking them keeps your content out of the assistants' answers as well as their training. If you want to be recommended by AI, allow at least the search agents.