Home / Research
Research 2026-11-21

How AI-ready are
124 software companies?

We ran our open-source AI-readiness audit against 124 well-known software companies — CRMs, dev tools, AI labs, fintech, analytics. Short version: none of them block the search crawlers of OpenAI, Anthropic or Perplexity, most publish llms.txt, and the gaps that remain are in structured data, comparison pages and anything built for agents.

New: how AI-ready are 150 independent hotels and restaurants?

The headline numbers

87/100
Median AI-readiness score across the 117 sites we could read fully (mean 85). 42 scored an A.
0 of 119
block OpenAI's, Anthropic's or Perplexity's search crawlers or user-triggered fetchers from their homepage in robots.txt.
11 of 119
block at least one AI crawler — almost all of them training crawlers, led by ByteDance's Bytespider (7 sites).
76 of 117
publish a valid /llms.txt; 24 also publish /llms-full.txt.
14 of 117
advertise an MCP server at a well-known URL — the least-adopted signal we measure.

Nobody blocks the big AI search engines. Policies lag behind new bots.

35 of 119 sites name AI crawlers explicitly in robots.txt. OpenAI's three agents are named about equally often. Anthropic's are not: ClaudeBot (training) appears on more than twice as many sites as Claude-SearchBot and Claude-User, the newer agents that fetch pages for live answers. The same holds for Perplexity-User:

CrawlerSites naming it explicitlyPurpose
GPTBot
29 of 119training
OAI-SearchBot
24 of 119search
ChatGPT-User
26 of 119user
ClaudeBot
26 of 119training
Claude-SearchBot
11 of 119search
Claude-User
12 of 119user
PerplexityBot
27 of 119search
Perplexity-User
11 of 119user
Google-Extended
24 of 119training
MistralAI-User
3 of 119user

Unnamed agents fall back to the User-agent: * rules, which on these sites allow them — so nothing is broken today. But a policy that names one vendor's training crawler and not its search agent may predate the newer agents and is worth revisiting. The fix is cheap: name the search and user agents you want, on purpose (example).

The most deliberate policy we found

Figma's robots.txt treats AI agents as an allowlist. OpenAI, Anthropic and Perplexity's search and user agents get about 9,000 explicit Allow lines — blog posts, font pages, the resource library, customer stories, solutions, templates, pricing and contact pages, across locales — followed by Disallow: /. Anything not on the list is off-limits. GPTBot and ClaudeBot (training) may fetch only /pricing/ and /pricing-faq/. Whatever you think of the trade-off, it reads as a choice made on purpose: training crawlers get Figma's prices, and little else.

The named-group trap: 15 of 36 sites

Under RFC 9309 a crawler that matches a named group ignores User-agent: * completely. So when a site writes a group for GPTBot or Claude-User and doesn't repeat its * Disallow lines, those paths are open to that crawler. Of the 36 sites whose robots.txt names at least one of the 26 agents we track, 15 leave a named group outside at least one of their own * Disallow paths; 11 of them involve AI-specific agents rather than Googlebot or Bingbot. Some are deliberate. Examples: on chargebee.com, 13 named AI agents may fetch /online-billing/, which the * group disallows; on redis.io, GPTBot, ChatGPT-User and four others may fetch /docs/staging/ and /studio/.

Check yours with the robots.txt tester, or fail the build on it with earncitations robots.

Where the gaps are

CheckSites failing
Answer-relevant schema types
36 of 117no Product/SoftwareApplication, FAQPage or similar
llms.txt missing or malformed
26 of 117
No Organization schema
20 of 117
No comparison / alternatives page
18 of 117found from the homepage or sitemap
No JSON-LD at all
17 of 117on any audited page
Prices not machine-readable
8 of 11765 more only partly
No pricing page linked
5 of 117
Homepage text needs JavaScript
2 of 117

The pattern: the basics of being readable are nearly solved (only 2 sites served an almost empty HTML shell; 110 link a pricing page). Being understood — schema that says what you sell and for how much, a fair page comparing you to alternatives — is where a third of the market still has work to do.

Structured prices vs the page

Offer markup states a price in a form software can read without guessing, so when the markup is missing a price or disagrees with the page, anything that reads the markup gets it wrong. We re-crawled the sample at 2026-11-21 14:38 UTC and looked at the 44 readable sites with Offer markup.

18 of 44 mark up only a price of 0, the free plan: software reading the markup learns that a free plan exists and nothing about the paid ones.

Of the rest, 23 could be compared with the prices printed in the pricing page's HTML, which is what AI crawlers read: Vercel found that none of the major AI crawlers it measured render JavaScript (Applebot aside). 19 match, counting monthly/annual equivalents and both sides of a billing toggle. 4 state a price the page's HTML doesn't show. One is plainly out of step: the markup on QuickBooks' plan comparison page prices its four QuickBooks Online plans at $17.50, $32.50, $49.50 and $117.50, while its pricing page lists $38, $85, $140 and $340 (or $11.40 to $102 with a 70%-off offer for three months). Of the other three, one is a price calculator's maximum and two are prices we couldn't find in the HTML; they may be loaded by script. The remaining 3 couldn't be compared: their pricing pages showed our crawler another currency, or no prices in the HTML.

The audit now runs this check on every site, the free AI price check shows both sides for yours, and watch alerts when a price change leaves the markup behind.

Do they answer their own buyers' questions?

A site can be perfectly readable and still not answer what buyers ask. On 2026-11-21 19:23 UTC we ran the free question map on every site in the sample: it proposes the questions a site's own pages suggest (its price, what it is, a free trial, an API if it publishes docs, SOC 2 if it has a security page, one "vs" question per comparison page, the integrations it lists) and checks each with the question doctor, reading raw HTML the way AI crawlers do. 116 of 124 sites could be read; 799 questions in all.

QuestionTimes askedAnsweredWeak pageMissingCrawl problem
How much does it cost?11663%33%2%3%
What is it?11670%28%2%1%
Is there a free trial?11623%72%4%1%
Does it have an API?9756%41%2%1%
Is it SOC 2 compliant?7016%81%1%1%
"X vs Y" (a comparison page they have)17277%17%5%1%
Does it integrate with Y? (an integration they list)8366%29%5%0%
"Best X for Y" (what their title claims)2993%7%0%0%

Answered: a page names the question (a heading or FAQ entry) and the first sentence under it answers, with the specific a buyer needs (a price, a yes or no). Weak: a page covers the topic but an assistant has to dig. Missing: no page we read mentions the question's key terms. Across all 799 questions, 58% were answered plainly and 38% were weak: the usual fix is one heading and one sentence on a page that already exists, not a new page.

Two gaps stand out. Free trials: only 27 of 116 sites answer "is there a free trial?" plainly. Most offer a free plan or credits and never say whether there's a trial, so an assistant has to guess. SOC 2: 70 sites have a security or trust page, but only 11 name SOC 2 in a heading and answer under it; on the rest an assistant has to dig through a list of certifications. Comparison pages hold up well: 77% of "X vs Y" questions land on a page that compares the two and says so.

Caveats: the doctor is a heuristic. It reads up to eight pages per question in the homepage's language (including the comparison, integration, API or security page that suggested the question, and pages a site serves to crawlers as Markdown), matches words rather than meaning, and can pick a weaker page when a site has no dedicated one (for example, prices shown only on the homepage). Except in the SOC 2 re-run, it read only the site's own host, so API docs on a separate docs subdomain weren't counted; it now also reads the company's docs, developer or help subdomain when the site links to it. Price, "what is", comparison, integration and best-for questions weren't re-asked after the blog-post rule below, so a few of their "answered" verdicts may rest on a blog post. A "weak" verdict means our crawler had to infer the answer, not that the site is wrong. A first run earlier the same night used a version of the doctor with bugs we found by checking its verdicts site by site ("Does it have an API?" came out weak on almost every site because "have" counted as a key word); it was discarded, and this table is from the corrected run. The doctor keeps improving; the run file is a snapshot of its verdicts at the time shown, and re-running tools/research/saas-questions.mjs gives current ones. "Times asked" counts questions, so a site with three comparison pages adds three "vs" questions. API questions re-asked after a fix found by spot-checking: a blog or news post can no longer count as the answer (Docker's 'does it have an API' had pointed to a blog post); re-asked 2026-11-21 19:58 UTC. Free-trial questions re-asked after two fixes found by spot-checking: a blog or news post can no longer count as the answer, and a FAQ heading of several sentences is judged by the question it actually asks (Paddle's 'I offer a subscription that does not have a free or promotional trial. Does this apply to me?' had counted as answering whether Paddle has a free trial); re-asked 2026-11-21 20:01 UTC. SOC 2 questions re-asked after a spot-check found two of six answered ones were a blog announcement (Docker) and a general article about SOC 2 CRMs (Folk): a blog post or article no longer counts as the answer, and a FAQ heading of several sentences is judged by the question it actually asks. This re-run also read the company's docs, developer or help subdomain when the site links to it; re-asked 2026-11-21 20:07 UTC. Run file: tools/research/runs/saas-questions-2026-11-22/saas-questions.json (question types and verdicts only).

Do they keep a Markdown copy for AI agents?

Some sites hand AI agents a plain Markdown version of a page, either linked from it (<link rel="alternate" type="text/markdown">) or sent when a client asks for Markdown (Accept: text/markdown). It's an honest, cheap way to be easy to read, but a second copy can leave out what the page shows people. On 2026-11-21 19:48 UTC we audited the sample again and, for each pricing page we found, asked for its Markdown copy the way an agent would and checked whether the copy has every price the page shows people. (A copy that lists more prices than the HTML isn't counted against it: that's often a table the page only loads with JavaScript.)

Software sitesCount
Readable, with a pricing page we found110
Keep a Markdown copy of it for agents34
Copy has every price the HTML shows21
Copy leaves out prices the HTML shows13

34 of 110 pricing pages (31%) answered an agent that asks for Markdown with a Markdown copy, more than we expected. In 13 of those 34, the copy leaves out prices people see on the page, and sometimes they're the plan prices themselves. We checked three by hand on 2026-11-21: Asana's copy has none of the per-user plan prices its HTML shows, Cloudflare's /plans copy leaves out its Pro and Business prices, and Netlify's leaves out add-on prices. An agent reading those copies can't quote what a plan costs. If you serve a copy, generate it from the same source as the page so they change together, and check that interactive parts (monthly/annual toggles, plan cards) come through. The free crawler view compares the two for any page, the audit checks your pricing page's copy, watch alerts when they drift, and the fix kit drafts a copy from your pricing page's own text.

Method: the audit (robots.txt first, our identified user agent) finds the pricing page from the homepage's links and sitemap; "no pricing page" sites (14) and unreadable ones (0) are left out. A copy counts when the server answers with text/markdown. Prices are compared as amounts with their currency ("USD 29" and "$29" are the same). Run file: tools/research/runs/md-copies-2026-11-22/md-copies.json (hosts and verdicts only).

Would a fix on the pricing page reach AI crawlers?

Jam's method starts with a rule we agree with: before reading what AI answers say after a change, check that the change was delivered. On 2026-11-21 21:27 UTC we ran our free Delivery check on /pricing for 123 sites in the sample, with robots.txt read first. 12 have no page there, which leaves 111 live pricing pages. For each, the question is whether an edit to it would reach the crawlers behind AI answers.

Of 111 live /pricing pagesCount
Canonical points to itself104
Canonical points to another page2
Marked noindex on the copy served to us1
Blocked in robots.txt for at least one AI search or user agent3
Listed in the sitemap93
Listed, with a lastmod date56
Linked from the homepage HTML (no JavaScript)99
Linked from llms.txt (of 89 sites with one; at least, see below)52+

The basics mostly hold. 104 of 111 pricing pages declare themselves canonical, and 99 are linked from the homepage's HTML. The gaps are in signalling change: 37 of the 93 listed in a sitemap have no lastmod, so the sitemap can't tell a crawler the page changed (Google uses lastmod only when it's consistently accurate, and Bing schedules crawls by it; sources on the Delivery check page). Of 89 sites with an llms.txt, 37 don't link their pricing page from it, as we counted on the day. That count is too high: the check then missed links to a page's Markdown copy (/pricing.md), which llms.txt files often use. vercel.com links one, and we haven't re-checked the others, so read 52 as a floor. The Delivery check counts these links now.

A lastmod only helps if it's the date the page changed. On 2026-11-21 21:56 UTC we re-checked the 56 pricing pages listed with one, and flagged a sitemap file where at least 90% of the URLs with a lastmod (20 or more) share the pricing page's date. 9 of 56 were: apollo.io, chargebee.com, clickup.com, cloudflare.com, cursor.com, docusign.com, figma.com, plaid.com, shopify.com. In all but one, every dated URL in the file carried the same date (ClickUp: 1,182 of 1,237), in files of 145 to 4,301 URLs. That is most likely the date the sitemap was generated, so it can't tell a crawler which page changed, and Google says it uses lastmod only when it's consistently accurate. The Delivery check now warns about this.

We checked the exceptions by hand. Coda: /pricing now redirects to Superhuman's site and adds a ?source= parameter. That site's robots.txt disallows every URL with a query string (Disallow: /*?), so the destination is blocked for every AI agent we track, even though the clean path is allowed. The fix is the redirect, not robots.txt. The Delivery check now says so when only the query string is blocked. OpenAI: /pricing redirects to chatgpt.com, whose robots.txt disallows PerplexityBot. Its Perplexity‑User group is written with a non-breaking hyphen (U+2011), so it matches no crawler; see names that match nothing below. Notion disallows Amzn-SearchBot. HubSpot and Wix point /pricing's canonical at another page (a product pricing page and /plans), so a fix belongs on that page. The one noindex was the regional copy that a location-based redirect sent our crawler to (noindex,nofollow,noarchive in its meta robots). Visitors elsewhere may get a different copy.

Method: one Delivery check per site, two at a time with a pause between, our identified user agent, and anything robots.txt disallows for us skipped. Pipedrive was left out because it had asked us to slow down earlier that day. No expected text was given, so the "in the HTML" step wasn't run. The sitemap step reads at most 8 sitemap files per site, likeliest first: 7 pages weren't found in what we read and 11 couldn't be settled: either the site's sitemap index has more files than the 8 we read, or its sitemap was gzipped, which the check couldn't read on that day and now can. Sites whose pricing lives at another path count as "no /pricing". Results: tools/research/runs/delivery-2026-11-22/results.json (hosts and verdicts only).

Could a coding agent set these products up from their docs?

Developers now ask Claude Code, Cursor or Codex to add a product to their app, and the agent follows the product's public docs. Jam lists "agent experience" as coming; our free docs check measures the part we can without running an agent. On 2026-11-21 23:59 UTC we ran it from the homepage of each of the 123 sites in the sample. We found developer docs from 109 of them (7 link none we could find; 5 didn't load for our crawler and 2 asked it to slow down, so we stopped).

Of 109 sites with docs we foundPassGapNot checked
Readable by AI crawlers (text without JavaScript, not behind sign-in, robots.txt allows)10180
A quickstart inside the docs94150
Prerequisites stated553915
Code as text in the quickstart's HTML771715
An install command or first request544015
Says how to authenticate742015
Shows what success looks like454915
A machine-readable API contract (OpenAPI)26830
An errors or troubleshooting page linked59500
A version or changelog named82270
llms.txt links the quickstart69400

1 of 109 pass all twelve checks; the median passes 8. "Not checked" means an earlier step failed (no quickstart, so nothing to read on it).

We checked the results by hand before publishing, and twice the check, not the sites, was wrong. Two earlier full runs were discarded and the check fixed. It had taken a homepage "API" link to a marketing page with a "Get started" heading for the docs. It missed a "Docs" link whose name ended in hidden "(opens in a new window)" text. Help centres outranked "API documentation" links. It didn't follow docs on a domain of the same name (shopify.dev, dropboxapi.com). And it stopped at the first of several quickstarts when that one had no code. An independent review of those fixes found two more bugs, fixed before the final run. What held up when we read the pages ourselves:

Method: one docs check per site, starting from the homepage, two sites at a time with a pause between, our identified user agent, and robots.txt read before every page, so anything it disallows for us or for the AI agents was skipped. Pipedrive was left out because it had asked us to slow down earlier that day. The check follows the homepage's docs links (up to four candidates, including a docs domain of the same name such as shopify.dev), then reads the docs' main page, the quickstart (up to four pages it links to if it is only a hub, and up to two other quickstarts if the first has no code), the API reference, OpenAPI and llms.txt, without JavaScript, within 45 seconds per site. It's a heuristic: it matches words in the page's own content, so it can miss a quickstart with unusual wording or credit a page that only mentions a key. It reads one quickstart; a product with many may do better or worse on another. On 2026-11-22 00:03 UTC we re-checked 27 of them after a fix: a link to a same-name domain (outreach.io redirects to outreach.ai) no longer counts as docs unless its text says docs. The table counts the re-checked rows. On 2026-11-22 00:11 UTC we re-checked 1 of them after a fix: a link to an "Error monitoring" or "Error tracking" product page no longer counts as the errors page. The table counts the re-checked rows. Results: tools/research/runs/docs-2026-11-22/results.json (hosts and verdicts only).

robots.txt names that match no crawler

Crawlers match a robots.txt group by name, as plain ASCII. A dash pasted from a word processor or a chat app looks right and matches nothing. chatgpt.com's robots.txt has a Perplexity‑User group written with a non-breaking hyphen (U+2011; we checked the raw bytes on 2026-11-22), so its rule applies to no crawler. To see how common this is, on 2026-11-21 we read the robots.txt of 123 sites in the sample (115 readable) and checked every User-agent name. We looked for lookalike or invisible characters and for names a missing, extra or swapped letter away from an AI agent we track. None had one. The first run flagged QuickBooks' xAI-SearchBot as a typo of OAI-SearchBot. It's a different crawler, so the check no longer counts a substituted letter as a typo. Rare, then, but silent when it happens. The audit, the robots.txt tester, earncitations robots and the MCP tool check_robots now flag it.

Top 10

#SiteScoreGrade
1clerk.com95A
2monday.com94A
3railway.com94A
4chargebee.com94A
5clickup.com93A
6calendly.com93A
7neon.com93A
8fly.io93A
9mixpanel.com93A
10resend.com93A

We publish the top of the table only. A low score from an outside crawler can reflect what that crawler was served rather than what AI crawlers see, so we don't single out individual low scorers. Every site's full report is reproducible with the open-source audit.

Method and caveats

Corrections

2026-11-21 (UTC). The audit graded 7 software companies in this sample as local businesses, so their scores above include checks for opening hours, a street address and phone number, and LocalBusiness markup that don't apply to them. The cause was a bug in how the audit decides what kind of business a site is: an address and phone number in Organization markup (which most companies publish), LocalBusiness markup for an office on an inner page, or an empty LocalBusiness template on the homepage (no name, opening hours of just commas) counted as signs of a local business. We rewrote that rule, tested it on restaurants, clinics, gyms, stores and software companies, and re-ran the whole sample on 2026-11-21. The re-scan's figures also reflect changes to the sites and checks added since, so per-site differences aren't all due to the fix.

SitePublished (profile, score)Re-scan 2026-11-21
atlassian.comlocal, 90software, 91
box.comlocal, 82software, 78
datadoghq.comlocal, 91software, 87
okta.comlocal, 88software, 89
squarespace.comlocal, 87software, 90
semrush.comlocal, 90software, 91
intercom.comlocal, 87software, 92

Across the sample, the re-scan's median score is 87 (published: 87) and the mean 85 (published: 85); grades A/B/C/D/F are 44/48/17/5/3 (published: 42/49/18/5/3). The new rule also reads developer-console and docs links, so 2 sites the published scan called general are now profiled as software: anthropic.com (general → software), groq.com (general → software). The tables above still show the published scan. Data: tools/research/runs/saas-2026-11-22/profile-correction.json.

Where does your site land? Run the same audit — free, about ten seconds.