Free tool
AI crawler log analyzer
Paste lines from your access log and see which AI crawlers came, what they read and where they were turned away.
It opens on an example. Replace it with yours: nothing you type leaves this browser.
Paste lines from your server or CDN log. Any format that prints the user agent works.
4 AI crawler visits from 3 crawlers
5 log lines read.
| Crawler | Company | What it's for | Requests | Errors | Most requested |
|---|---|---|---|---|---|
| ClaudeBot | Anthropic | Training | 2 | 0 | /blog/launch-guide |
| GPTBot | OpenAI | Training | 1 | 0 | /pricing |
| PerplexityBot | Perplexity | AI search | 1 | 1 | /old-page |
- Fix
1 of their requests got an error (4xx or 5xx). A 403 usually means a firewall or bot rule is turning the crawler away.
Anyone can send a crawler's name. To be sure a visit is real, check its IP against the ranges the company publishes.
The log never leaves your browser.
How do I know if AI crawlers visit my site?
AI crawlers don't run JavaScript, so they never appear in analytics. They do appear in your server or CDN access log, under user agents such as GPTBot, ClaudeBot and PerplexityBot.
What it checks
Every AI crawler we know, found by its user agent
Requests per crawler, and whose crawler it is
What each collects for: training, search or a person's request
Errors (4xx and 5xx) that mean a crawler was turned away
The page each one asked for most
Use it from your AI assistant
Claude, ChatGPT, Cursor and any other MCP client can run this for you. Add the server below, then ask for it in your own words: the tool is read_ai_crawler_log. No sign-up and no key.
https://api.makerradar.io/mcpOr call the REST API, which answers the same report as JSON:
curl -X POST https://api.makerradar.io/v1/tools/ai-crawler-log-analyzer \
-H "content-type: application/json" -d '{"log":"203.0.113.7 - - [14/Nov/2026:09:12:01 +0000] \"GET /pricing HTTP/1.1\" 200 5120 \"-\" \"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\"\n203.0.113.9 - - [14/Nov/2026:09:12:44 +0000] \"GET /blog/launch-guide HTTP/1.1\" 200 18230 \"-\" \"Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\"\n203.0.113.9 - - [14/Nov/2026:09:13:02 +0000] \"GET /blog/launch-guide HTTP/1.1\" 200 18230 \"-\" \"Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\"\n198.51.100.4 - - [14/Nov/2026:09:14:10 +0000] \"GET /old-page HTTP/1.1\" 404 312 \"-\" \"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\"\n192.0.2.20 - - [14/Nov/2026:09:15:31 +0000] \"GET / HTTP/1.1\" 200 9120 \"https://example.com/\" \"Mozilla/5.0 (Macintosh) Safari/605.1.15\""}'Get a page that passes, free
Every product on MakerRadar gets a permanent page with a clean title and description, structured data, an FAQ, a Markdown version for AI assistants and a place in our llms.txt — and a backlink you can earn.
Related tools
GA4 AI traffic setup
Copy one pattern and follow three steps to see how many visitors AI assistants send you.
Runs in your browser
AI crawler robots.txt generator
Choose, crawler by crawler, which AI companies may read your site for search and which for training, and copy the robots.txt lines.
Runs in your browser
AI visibility checker
Find out whether AI assistants and answer engines are allowed to read your site — and can understand it when they do.
Reads the page
robots.txt tester
Paste any URL on a site and see, for Google, Bing and every major AI crawler, whether its robots.txt lets them fetch it — and which line decides.
Reads the page
More: ai & agents
Frequently asked questions
Where do I find my access log?
On your server (nginx or Apache access.log), or in your host's or CDN's dashboard: Cloudflare, Vercel and Netlify all offer request logs.
Why would a crawler get a 403?
A firewall, bot protection or rate limit is refusing it. If you want to be in AI answers, allow the search crawlers through.
Can a user agent be faked?
Yes. Anyone can send GPTBot's name. To be sure, compare the visitor's IP with the ranges the company publishes.
Is what I type stored?
No. It runs in your browser: nothing you type or paste is sent to us or kept.