# EarnCitationsAudit — our crawler · EarnCitations

> What the EarnCitationsAudit user agent is, what it fetches, how it honours robots.txt, and how to block it.

Source: https://earncitations.com/bot.html

> Every page on this site: https://earncitations.com/llms.txt (the whole site as one file: https://earncitations.com/llms-full.txt)

Crawler updated 2026-11-22

## EarnCitationsAudit

If you saw EarnCitationsAudit/0.1 (+https://earncitations.com/bot.html) in your logs, someone ran an AI-readiness audit of your site — either a person using the free audit, or our published research on how ready websites are for AI search.

### What it fetches

One audit is a small, one-off set of requests, not a crawl:

- The page that was asked for (usually your homepage).

- /robots.txt, /sitemap.xml, /llms.txt, /llms-full.txt, /.well-known/facts.json, /.well-known/facts.pub, /.well-known/mcp.json, /.well-known/mcp and /.well-known/ard.json. If your robots.txt (Agentmap:), homepage (<link rel="ard">) or its Link header points to an ARD manifest elsewhere on your site, that file too (two at most, where robots.txt allows us). With none of those, the old name /.well-known/ai-catalog.json. When you ask us to audit a section (/docs/) and there's no root llms.txt, that section's /docs/llms.txt.

- Up to six of your public pages (such as pricing, about, FAQ and docs), only if robots.txt allows them for EarnCitationsAudit. For the pricing page, also its Markdown copy for agents if it has one: the copy it links to, or the same URL asked for as Markdown.

- Up to eight of the links in your llms.txt, on your own site and only where robots.txt allows them, to check they still load (64 KB read from each).

For the audit, responses are capped at 2 MB, each request times out after 8 seconds and the whole audit after 25. It doesn't run JavaScript, submit forms, log in, or follow links off your domain. Results for the same site are cached for ten minutes.

### Other checks with the same user agent

Our other free tools send the same EarnCitationsAudit user agent. Each is started by a person asking about a page or a site; none crawls on its own. None of them runs JavaScript, submits forms or logs in. Responses are capped at 3 MB (5 MB for the delivery check), with at most 15 seconds per request.

| Check | What it fetches | robots.txt |
| --- | --- | --- |
| [Question doctor](https://earncitations.com/question.html), [question map](https://earncitations.com/map.html) | The homepage, robots.txt, the sitemap and up to six pages that might answer each question. A docs or help subdomain of the same domain is read too when the site links to it. | Read first, including each subdomain's own; disallowed pages are skipped. |
| [Docs check](https://earncitations.com/docs.html) | The homepage, then the developer docs, which can be on a docs subdomain or the docs' own domain (shopify.dev from shopify.com). About twenty-five requests within 45 seconds. For the npm packages a quickstart installs (four at most), one request each to npm's public registry. | Read before every docs page; disallowed pages are skipped. |
| [Crawler view](https://earncitations.com/view.html) | The one page that was asked for, and robots.txt, to say which AI agents it blocks. If the page has a Markdown copy for agents, that copy too (the one it links to on the same host, or the same URL asked for as Markdown), to compare the two. | Fetched alongside the page, because a person asked for that page. |
| [Delivery check](https://earncitations.com/delivery.html) | The page that was asked for, robots.txt, the sitemap (eight files at most), the homepage and llms.txt, plus for a page in a folder the nearest two folder-level llms.txt files (/docs/llms.txt for /docs/setup) and a Markdown copy the page advertises, robots.txt permitting. About sixteen requests within 45 seconds. | Read to report whether AI crawlers may fetch the page. |
| [Answer check](https://earncitations.com/claims.html) | The pages an AI answer cites, eight at most. | Read first for each site; disallowed pages aren't read. |

Our published research identifies itself as EarnCitationsResearch/0.1, with a link to this page. Everything we keep, and for how long, is on [Data and security](https://earncitations.com/security.html).

### robots.txt

We never fetch subpages that your robots.txt disallows for EarnCitationsAudit (or for * when there's no group for us). For batch research we check robots.txt first and skip your site entirely if the start page is disallowed. To opt out completely:

```
User-agent: EarnCitationsAudit
Disallow: /
```

A person who runs a one-off audit of your homepage from our site is treated like a visitor checking your page: we fetch the page they asked for and the well-known files above, which are meant to be public.

### Contact

Problems or questions: [hello@earncitations.com](mailto:hello@earncitations.com).
