Glossary

robots.txt

robots.txt is a file at a site's root telling crawlers which paths they may fetch. It controls crawling, not indexing.

It's also where a site states its stance on AI crawlers. MakerRadar's robots.txt welcomes search and AI crawlers alike, so listings can be found in search and cited in AI answers.

See it on MakerRadar

How agents may use our content

Related terms

SitemapA sitemap is a file listing a site's pages, usually /sitemap.xml, so search engines can find and recrawl every page worth indexing.
llms.txtllms.txt is a Markdown file at a site's root that tells AI models what the site is and links its key pages, in a form they read easily.

Launch on MakerRadar for a permanent, search-friendly page and your first backlinks.

Launch on MakerRadar