Skip to content
MakerRadar

Our crawler

MakerRadarBot

MakerRadarBot is MakerRadar's web crawler. It reads the websites of products listed on MakerRadar — to fill in a listing, check who owns a site, and make sure we never send visitors somewhere unsafe. It names itself honestly, signs every request, and stays light: most sites see it rarely, if at all.

On this page
  1. How to recognise it
  2. What it fetches, and why
  3. Controlling it with robots.txt
  4. Contact

How to recognise it

Every request carries this user agent:

Mozilla/5.0 (compatible; MakerRadarBot/1.0; +https://makerradar.io/bot)

and a Web Bot Auth signature (RFC 9421): a signature-agent header naming this site, and signature-input / signature headers signed with our Ed25519 key (tag="web-bot-auth", valid for 2 minutes). Anyone can check it with the public key at https://makerradar.io/.well-known/http-message-signatures-directory; a request without a valid signature isn't ours, whatever its user agent says.

What it fetches, and why

  • Filling in a listing its maker submits. When a maker submits their product, the page they give us — its title, description, logo and web app manifest — to draft the listing and copy the logo to MakerRadar's own storage. How often: once per submission. The site's own maker or owner asked for it.
  • Checking an ownership claim. The verification file or homepage meta tag a person placed on their site to prove they own it. How often: only when they ask us to check. The site's own maker or owner asked for it.
  • Checking a launch badge. The homepage of a listing whose maker embedded our badge, to confirm it's still there. How often: about every 7 days. The site's own maker or owner asked for it.
  • Keeping visitors safe (link health). The landing page of each listed site, to notice one that has gone, now redirects to another site or changed hands — so we stop sending visitors there. How often: once a day per site. A safety check of a page we already link to.
  • Reading a site nobody here owns. The homepage of a product someone declared as a competitor, or one our discovery found, before our team reviews it — its name, a short excerpt and its logo; never its copy. How often: once a day per site. It reads robots.txt first.
  • Discovering new launches. Show HN and a few curated launch feeds, then the product pages they link to, to suggest them to our team. How often: once a minute per site, at most 50 visits a run. It reads robots.txt first.
  • Free SEO and AI-visibility tools. The page a visitor types into one of our free checkers, with its robots.txt and llms.txt. How often: limited per site, however many visitors ask. It reads robots.txt first.
  • Documents an app or agent points us to. An OAuth client's metadata document, or an image an AI agent asks us to add to its person's listing — the exact URL given, nothing else. How often: once per request. Only the URL it was given, when it's asked for.

It never fills in forms, logs in, follows links across your site or downloads more than it needs: each fetch is one page (or one file), capped in size and time.

Controlling it with robots.txt

MakerRadarBot reads robots.txt under its own name, then *. To keep it out of your site:

User-agent: MakerRadarBot
Disallow: /

That stops discovery, our free tools and any read of a site nobody here owns. A fetch the site's own maker or owner asked for — a submission, an ownership check, their badge — and the daily safety check of a page we link to still happen; if you'd rather not be listed at all, write to us and we'll take the listing down.

Contact

Questions about MakerRadarBot, or a request that looks wrong: support@makerradar.io. To report abuse: abuse-report@makerradar.io. More about who runs MakerRadar: About; how we treat crawlers that read our own pages: crawling policy.