Skip to content
MakerRadar

Free tool

robots.txt tester

Paste any URL on a site and see, for Google, Bing and every major AI crawler, whether its robots.txt lets them fetch it — and which line decides.

Any URL on the site: its path is tested against the site's robots.txt.

How do I check if Google or ChatGPT can crawl a page?

robots.txt is a text file at a site's root that tells each crawler, by its user agent, which paths it may fetch. A tester applies those rules to one URL for each crawler and shows the line that decides.

What it checks

  • Whether the site has a robots.txt we can read

  • Googlebot and Bingbot on the exact path you gave

  • 21 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended …), each with the rule that decides

  • Sitemap lines, Content Signals and Crawl-delay

  • Lines crawlers will skip: typos and rules before any User-agent

Use it from your AI assistant

Claude, ChatGPT, Cursor and any other MCP client can run this check for you. Add the server below, then ask it to check a site: the tool is test_robots_txt. No sign-up and no key.

https://api.makerradar.io/mcp

Or call the REST API, which answers the same report as JSON:

curl -X POST https://api.makerradar.io/v1/tools/robots-txt-tester \
  -H "content-type: application/json" -d '{"url":"https://example.com"}'

MCP server guide · API reference

Get a page that passes, free

Every product on MakerRadar gets a permanent page with a clean title and description, structured data, an FAQ, a Markdown version for AI assistants and a place in our llms.txt — and a backlink you can earn.

Related tools

  • AI visibility checker

    Find out whether AI assistants and answer engines are allowed to read your site — and can understand it when they do.

    Reads the page

  • Sitemap checker

    Paste your site's address: we find its sitemap the way search engines do, read it, and tell you what would stop them using it.

    Reads the page

  • AI crawler robots.txt generator

    Choose, crawler by crawler, which AI companies may read your site for search and which for training, and copy the robots.txt lines.

    Runs in your browser

More free tools

  • Website grader

    Paste a URL and get one grade for the whole site: found on Google, quoted by AI, shared well and safe to visit, with the fixes that matter most first.

    Reads the page

  • Launch checklist

    Everything a product's website needs before launch day, in order, each step with the free tool that checks it, then the steps for launch day itself.

    Runs in your browser

  • Launch directory planner

    Pick what you need (free, a dofollow link, a minimum Domain Rating) and get a short, ranked plan of where to launch, a few a week, with each directory's DR.

    Answers from MakerRadar's own catalogue

  • Domain availability checker

    Type a name and see which startup domains are free, and for the taken ones, when they were registered and with which registrar.

    Looks up the domain, never visits the site

  • Alternatives finder

    Paste a product's address and see the products listed here that do the same job, in the order its alternatives page ranks them.

    Answers from MakerRadar's own catalogue

  • SEO checker

    Paste a URL and see what Google reads on that page — and what to fix first.

    Reads the page

  • AI citation checker

    Paste a URL and see how easy that page is for ChatGPT, Perplexity and Google's AI Overviews to quote, signal by signal, with what to add first.

    Reads the page

  • What AI sees

    Paste a URL and read the page the way an AI assistant does: the HTML your server sends, turned into Markdown, with its outline, and whether your content only appears once JavaScript runs.

    Reads the page

  • llms.txt generator

    Paste your site's address: we check its /llms.txt against the spec, or draft one from your homepage that you can save as it is.

    Reads the page

  • Agent readiness checker

    Paste your site's address and see which of the files and signals AI agents look for it publishes: the same set this site publishes for them.

    Reads the page

  • Social card preview

    Paste a URL and see the card X, LinkedIn, Facebook and Slack will show when someone shares it — and why it might show no image at all.

    Reads the page

  • Logo & favicon grabber

    Paste a URL and see every icon the site declares, its real size, and the one a directory or app would pick: shown from the site's own address, never copied.

    Reads the page

  • Structured data checker

    Paste a URL and see every schema.org type the page declares in JSON-LD, which rich results it can earn, and the fields each one is missing.

    Reads the page

  • Broken link checker

    Paste a URL and see every link on that page that's broken or redirects: up to 50 links, same-site first, each with its status and anchor text.

    Reads the page

  • Redirect checker

    Paste a URL and see every hop it takes before it lands: the status code, the address and what changed, with loops and long chains flagged.

    Reads the page

  • SERP snippet preview

    Paste a URL and see how its page would show in Google, with where the title and description would be cut and how to fit them.

    Reads the page

  • Keyword generator

    Start from what your product is and get the searches — and the questions people ask AI — that lead to products like it.

    Runs in your browser

  • Domain rating checker

    See how strong a domain's backlink profile is — the number launch directories are ranked by.

    Looks up the domain, never visits the site

  • Security headers checker

    See which protections a site sends with every page — and which it's missing.

    Reads the page

  • Email DNS checker (SPF, DMARC, DKIM)

    Will mail from this domain land in the inbox? Check its SPF, DMARC and DKIM in one go.

    Looks up the domain, never visits the site

  • Domain age checker

    How old is a domain, who registered it and when does it expire — straight from the registry.

    Looks up the domain, never visits the site

  • SSL certificate checker

    Who issued the site's certificate, when it expires, and whether the zone allows that issuer.

    Looks up the domain, never visits the site

  • Hosting & CDN lookup

    Where a site is served from: its platform, its edge network and the origin behind it.

    Looks up the domain, never visits the site

  • Tech stack detector

    Find out what a site is built with and which third-party scripts it loads.

    Reads the page

Frequently asked questions

How does robots.txt decide?

A crawler uses the group that names it, else the User-agent: * group. Within that group the longest matching path rule wins, and Allow wins a tie (RFC 9309). The tester applies exactly that and shows you the deciding line.

Does blocking GPTBot keep me out of ChatGPT?

Only out of training. ChatGPT's search and browsing use OAI-SearchBot and ChatGPT-User, which have their own lines. The same split holds for Claude (ClaudeBot vs Claude-SearchBot and Claude-User) and Perplexity.

Is robots.txt a security control?

No. It's a request polite crawlers honour. Anything secret needs a login, and a Disallow line actually advertises the path to anyone who reads the file.