# Answer Index — methodology · EarnCitations

> How the Answer Index measures which brands AI assistants recommend: engines × buyer questions × repeated samples, with confidence intervals.

Source: https://earncitations.com/answer-index.html

> Every page on this site: https://earncitations.com/llms.txt (the whole site as one file: https://earncitations.com/llms-full.txt)

Why an indexStudy designMetricsUncertaintyLimitationsIntegrity rulesRun it yourselfCategories

Methodology v0.1 · first edition in preparation

## The Answer Index

Who does AI recommend? A public, reproducible benchmark of AI answer engines, category by category — with the questions, the raw answers and the harness all open.

### Why an index

"AI visibility" numbers often come from dashboards whose prompts, sampling and math aren't published, so nobody outside can check them. The Answer Index publishes all three: fixed public question sets, repeated sampling, published confidence intervals and an open-source harness (AnswerBench) anyone can run to reproduce or challenge a result.

### Study design

| Element | Choice |
| --- | --- |
| Questions | 12–40 buyer questions per category, written as real people ask them ("best X for Y", "X vs Y", "is X worth it for…"), reviewed for leading phrasing. Published with each edition. |
| Engines | Each engine through its official API, with web search on where available. Model IDs and dates are recorded per answer. |
| Sampling | Every question is asked ≥3 times per engine in fresh conversations. Answers are non-deterministic; one sample is an anecdote. |
| Brands | A category roster with names, aliases and domains, published in advance. Mentions outside the roster are logged as "other". |
| Extraction | Deterministic, alias-aware, word-boundary matching — no LLM judge on the headline numbers, so results are auditable line by line. |

### Metrics

| Metric | Definition |
| --- | --- |
| Visibility | Share of answers that name the brand at least once. |
| Share of voice | The answers that name the brand, as a share of all roster brands' appearances in answers. Each brand counts once per answer, however often it is named. |
| First-mention position | Mean rank of the brand among brands named in an answer (1 = named first). Lower is better. |
| Recommendation rate | Share of answers where the brand is among the first three brands named. A position measure: it doesn't read the answer's wording. |
| Citation share | The brand's share of all the URLs the answers cite or retrieve: those on its domains or their subdomains. |
| Stability | How consistently the brand appears across repeated samples of the same question. |
| Source mix | Which domains the engines cite for the category — the supply chain of the answer. |

### Uncertainty

Every headline number ships with a 95% bootstrap confidence interval, resampling questions (not samples) with a fixed seed, so intervals reflect how much the result depends on which questions were asked. If two brands' intervals overlap, we don't call a winner.

### Limitations, stated plainly

- API ≠ app. Consumer apps add memory, personalization, location and different system prompts. API answers are a controlled proxy, not what every user sees.

- Time. Engines change weekly. Each edition is a dated snapshot.

- Question choice drives results. That's why the questions are published and intervals resample them.

- Mentions aren't endorsements. "Avoid X" still names X, and counts toward visibility. The harness doesn't score sentiment, so read the raw answers before treating a mention as a recommendation.

### Integrity rules

EarnCitations customers get no special treatment in the Index. Rosters, questions and raw answers are published before any customer sees their numbers, and the harness is open so anyone can re-run an edition.

### Run it yourself

```
# free, offline: synthetic answers to see the pipeline end to end
npm run bench -- run tools/bench/examples/demo-fictional.json --dry-run

# real engines (spends your API credits; asks for --yes)
npm run bench -- estimate tools/bench/examples/crm-agencies.json
npm run bench -- run tools/bench/examples/crm-agencies.json --engines anthropic --yes
```

### Planned categories for the first edition

CRM for small agencies · AI meeting notetakers · Developer observability · Payroll for startups · Project management for agencies · Running shoes · Skincare for sensitive skin · Family dentists by city · Accounting firms for freelancers · Coffee roasters with subscriptions.

Want your category included? [Tell us](mailto:hello@earncitations.com) — we'll publish the roster before we run it.
