Methodology v0.1 · first edition in preparation

The Answer Index

Who does AI recommend? A public, reproducible benchmark of AI answer engines, category by category — with the questions, the raw answers and the harness all open.

Why an index

"AI visibility" numbers often come from dashboards whose prompts, sampling and math aren't published, so nobody outside can check them. The Answer Index publishes all three: fixed public question sets, repeated sampling, published confidence intervals and an open-source harness (AnswerBench) anyone can run to reproduce or challenge a result.

Study design

ElementChoice
Questions12–40 buyer questions per category, written as real people ask them ("best X for Y", "X vs Y", "is X worth it for…"), reviewed for leading phrasing. Published with each edition.
EnginesEach engine through its official API, with web search on where available. Model IDs and dates are recorded per answer.
SamplingEvery question is asked ≥3 times per engine in fresh conversations. Answers are non-deterministic; one sample is an anecdote.
BrandsA category roster with names, aliases and domains, published in advance. Mentions outside the roster are logged as "other".
ExtractionDeterministic, alias-aware, word-boundary matching — no LLM judge on the headline numbers, so results are auditable line by line.

Metrics

MetricDefinition
VisibilityShare of answers that name the brand at least once.
Share of voiceThe answers that name the brand, as a share of all roster brands' appearances in answers. Each brand counts once per answer, however often it is named.
First-mention positionMean rank of the brand among brands named in an answer (1 = named first). Lower is better.
Recommendation rateShare of answers where the brand is among the first three brands named. A position measure: it doesn't read the answer's wording.
Citation shareThe brand's share of all the URLs the answers cite or retrieve: those on its domains or their subdomains.
StabilityHow consistently the brand appears across repeated samples of the same question.
Source mixWhich domains the engines cite for the category — the supply chain of the answer.

Uncertainty

Every headline number ships with a 95% bootstrap confidence interval, resampling questions (not samples) with a fixed seed, so intervals reflect how much the result depends on which questions were asked. If two brands' intervals overlap, we don't call a winner.

Limitations, stated plainly

Integrity rules

EarnCitations customers get no special treatment in the Index. Rosters, questions and raw answers are published before any customer sees their numbers, and the harness is open so anyone can re-run an edition.

Run it yourself

# free, offline: synthetic answers to see the pipeline end to end
npm run bench -- run tools/bench/examples/demo-fictional.json --dry-run

# real engines (spends your API credits; asks for --yes)
npm run bench -- estimate tools/bench/examples/crm-agencies.json
npm run bench -- run tools/bench/examples/crm-agencies.json --engines anthropic --yes

Planned categories for the first edition

CRM for small agencies · AI meeting notetakers · Developer observability · Payroll for startups · Project management for agencies · Running shoes · Skincare for sensitive skin · Family dentists by city · Accounting firms for freelancers · Coffee roasters with subscriptions.

Want your category included? Tell us — we'll publish the roster before we run it.