Skip to content
MakerRadar

Free tool

AI crawler log analyzer

Paste lines from your access log and see which AI crawlers came, what they read and where they were turned away.

It opens on an example. Replace it with yours: nothing you type leaves this browser.

Paste lines from your server or CDN log. Any format that prints the user agent works.

4 AI crawler visits from 3 crawlers

5 log lines read.

4 AI crawler visits from 3 crawlers
CrawlerCompanyWhat it's forRequestsErrorsMost requested
ClaudeBotAnthropicTraining20/blog/launch-guide
GPTBotOpenAITraining10/pricing
PerplexityBotPerplexityAI search11/old-page
  • Fix

    1 of their requests got an error (4xx or 5xx). A 403 usually means a firewall or bot rule is turning the crawler away.

  • Anyone can send a crawler's name. To be sure a visit is real, check its IP against the ranges the company publishes.

  • The log never leaves your browser.

How do I know if AI crawlers visit my site?

AI crawlers don't run JavaScript, so they never appear in analytics. They do appear in your server or CDN access log, under user agents such as GPTBot, ClaudeBot and PerplexityBot.

What it checks

  • Every AI crawler we know, found by its user agent

  • Requests per crawler, and whose crawler it is

  • What each collects for: training, search or a person's request

  • Errors (4xx and 5xx) that mean a crawler was turned away

  • The page each one asked for most

Use it from your AI assistant

Claude, ChatGPT, Cursor and any other MCP client can run this for you. Add the server below, then ask for it in your own words: the tool is read_ai_crawler_log. No sign-up and no key.

https://api.makerradar.io/mcp

Or call the REST API, which answers the same report as JSON:

curl -X POST https://api.makerradar.io/v1/tools/ai-crawler-log-analyzer \
  -H "content-type: application/json" -d '{"log":"203.0.113.7 - - [14/Nov/2026:09:12:01 +0000] \"GET /pricing HTTP/1.1\" 200 5120 \"-\" \"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\"\n203.0.113.9 - - [14/Nov/2026:09:12:44 +0000] \"GET /blog/launch-guide HTTP/1.1\" 200 18230 \"-\" \"Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\"\n203.0.113.9 - - [14/Nov/2026:09:13:02 +0000] \"GET /blog/launch-guide HTTP/1.1\" 200 18230 \"-\" \"Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\"\n198.51.100.4 - - [14/Nov/2026:09:14:10 +0000] \"GET /old-page HTTP/1.1\" 404 312 \"-\" \"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\"\n192.0.2.20 - - [14/Nov/2026:09:15:31 +0000] \"GET / HTTP/1.1\" 200 9120 \"https://example.com/\" \"Mozilla/5.0 (Macintosh) Safari/605.1.15\""}'

MCP server guide · API reference

Get a page that passes, free

Every product on MakerRadar gets a permanent page with a clean title and description, structured data, an FAQ, a Markdown version for AI assistants and a place in our llms.txt — and a backlink you can earn.

Related tools

  • GA4 AI traffic setup

    Copy one pattern and follow three steps to see how many visitors AI assistants send you.

    Runs in your browser

  • AI crawler robots.txt generator

    Choose, crawler by crawler, which AI companies may read your site for search and which for training, and copy the robots.txt lines.

    Runs in your browser

  • AI visibility checker

    Find out whether AI assistants and answer engines are allowed to read your site — and can understand it when they do.

    Reads the page

  • robots.txt tester

    Paste any URL on a site and see, for Google, Bing and every major AI crawler, whether its robots.txt lets them fetch it — and which line decides.

    Reads the page

More: ai & agents

Frequently asked questions

Where do I find my access log?

On your server (nginx or Apache access.log), or in your host's or CDN's dashboard: Cloudflare, Vercel and Netlify all offer request logs.

Why would a crawler get a 403?

A firewall, bot protection or rate limit is refusing it. If you want to be in AI answers, allow the search crawlers through.

Can a user agent be faked?

Yes. Anyone can send GPTBot's name. To be sure, compare the visitor's IP with the ranges the company publishes.

Is what I type stored?

No. It runs in your browser: nothing you type or paste is sent to us or kept.