# AI crawler directory: every AI bot, what it does, how to allow or block it · EarnCitations

> 26 AI crawlers and robots.txt tokens from 12 operators: what each does in the operator's own words, how it treats robots.txt, and copy-paste rules.

Source: https://earncitations.com/crawlers.html

> Every page on this site: https://earncitations.com/llms.txt (the whole site as one file: https://earncitations.com/llms-full.txt)

Directory checked 2026-11-21 against each operator's docs

## AI crawler directory

26 agents from 12 operators, in three kinds: answer engines that index you, fetchers that read you live when a user asks, and training crawlers. Each description paraphrases the operator's own documentation, linked in the last column. "Named by" is how many of the 119 software sites in [our research scan](https://earncitations.com/research.html) name the agent in robots.txt; agents added after that scan show a dash.

### Answer engines

Index pages that AI answers cite. Block one and your pages can drop out of the AI search it feeds.

| Agent | Operator | What it does (per the operator) | Named by | Source |
| --- | --- | --- | --- | --- |
| OAI-SearchBot | OpenAI | Surfaces websites in search results in ChatGPT's search features. | 24 | [docs](https://developers.openai.com/api/docs/bots) |
| Claude-SearchBot | Anthropic | Navigates the web to improve search result quality for Claude users. Blocking it may reduce your visibility in Claude's answers. | 11 | [docs](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results. | 27 | [docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| Googlebot | Google | Google Search's main crawler. | 6 | [docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |
| Bingbot | Microsoft | Bing's standard crawler; handles most of Bing's crawling. | 11 | [docs](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) |
| Applebot | Apple | Powers search in Spotlight, Siri and Safari; its data may also ground AI answers in Siri and Search. Follows your Googlebot rules if Applebot isn't named. | 2 | [docs](https://support.apple.com/en-us/119829) |
| Amzn-SearchBot | Amazon | Makes content eligible for search experiences such as Alexa. Not used for generative AI training. | — | [docs](https://developer.amazon.com/docs/amazonbot/amazonbot) |
| DuckAssistBot | DuckDuckGo | Crawls pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. | 8 | [docs](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) |
| MistralAI-Index | Mistral | Indexes content for Mistral search, used to answer questions in Vibe. Not used for training. | — | [docs](https://docs.mistral.ai/robots) |

### User-triggered fetchers

Load a page live when someone asks an assistant about it. Not crawlers in the usual sense; several operators say they may not follow robots.txt.

| Agent | Operator | What it does (per the operator) | Named by | Source |
| --- | --- | --- | --- | --- |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT or Custom GPT user's request needs it. Not used for automatic crawling or to decide what appears in search. | 26 | [docs](https://developers.openai.com/api/docs/bots) |
| Claude-User | Anthropic | Fetches pages when someone asks Claude a question. Blocking it stops Claude retrieving your content for those users. | 12 | [docs](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Perplexity-User | Perplexity | Fetches pages for user actions in Perplexity. Perplexity says it generally ignores robots.txt because a user requested the fetch. | 11 | [docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| Google-Agent | Google | Used by agents hosted on Google infrastructure to navigate the web and perform actions at a user's request. Google says its user-triggered fetchers generally ignore robots.txt. | — | [docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) |
| Google-GeminiNotebook | Google | Fetches individual URLs that Gemini Notebook users add as sources (it replaces Google-NotebookLM, which Google lists as supported until August 2026). Google says its user-triggered fetchers generally ignore robots.txt. | — | [docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) |
| Meta-ExternalFetcher | Meta | Fetches individual links at a user's request, including for AI agents completing tasks. Meta says it may bypass robots.txt. | — | [docs](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) |
| Amzn-User | Amazon | Fetches live information for user requests such as Alexa questions. May not follow all robots.txt directives. Not used for training. | — | [docs](https://developer.amazon.com/docs/amazonbot/amazonbot) |
| MistralAI-User | Mistral | Fetches pages for user actions in Vibe. | 3 | [docs](https://docs.mistral.ai/robots) |

### Training crawlers and tokens

Collect content for model training. Blocking them is a legitimate choice; future models will just know less about you.

| Agent | Operator | What it does (per the operator) | Named by | Source |
| --- | --- | --- | --- | --- |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's generative AI foundation models. Independent of OAI-SearchBot: blocking it doesn't remove you from ChatGPT search. | 29 | [docs](https://developers.openai.com/api/docs/bots) |
| ClaudeBot | Anthropic | Collects web content that could contribute to Anthropic's model training. Blocking it excludes your future content from training datasets. | 26 | [docs](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Google-Extended (token) | Google | A robots.txt token, not a crawler. Controls whether content Google crawls may train future Gemini models and ground answers in Gemini Apps and Vertex AI. No effect on Google Search. | 24 | [docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |
| Applebot-Extended (token) | Apple | A robots.txt token, not a crawler. Disallow it to opt out of training Apple's foundation models; Applebot still crawls for search. | 16 | [docs](https://support.apple.com/en-us/119829) |
| CCBot | Common Crawl | Builds Common Crawl's open web archive, which many AI training datasets draw on. | 21 | [docs](https://commoncrawl.org/ccbot) |
| Meta-ExternalAgent | Meta | Crawls for use cases such as training foundation AI models or improving products (Meta's wording). | 15 | [docs](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) |
| Bytespider | ByteDance | ByteDance's crawler. We found no official documentation of its purpose or robots.txt behaviour; classed as training based on public reporting. | 14 | none found |
| Amazonbot | Amazon | Used to improve Amazon products and services; Amazon says it may be used to train Amazon AI models. | 12 | [docs](https://developer.amazon.com/docs/amazonbot/amazonbot) |
| MistralAI-Training | Mistral | Crawls web content to build datasets for training Mistral models. | — | [docs](https://docs.mistral.ai/robots) |

### Other documented agents

Operators document these next to their AI crawlers, but they don't index you for answers, fetch for a user, or train models, so the audit doesn't score them. If you name them in robots.txt, the usual rules apply.

| Agent | Operator | What it does | Source |
| --- | --- | --- | --- |
| OAI-AdsBot | OpenAI | Visits landing pages submitted as ads on ChatGPT to check them against OpenAI's policies, and may use their content to decide when to show the ad. OpenAI says it only visits pages submitted as ads and its data isn't used to train foundation models. [IP ranges](https://openai.com/adsbot.json). | [developers.openai.com](https://developers.openai.com/api/docs/bots) |
| Google-CloudVertexBot | Google | Crawls a site when its owner asks for it while building Vertex AI Agents. Google says the token has no effect on Google Search or other products. | [developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |

### Changes

Operators add and repurpose agents often. Every change to this list is dated here and in the [open dataset (crawlers.json)](https://earncitations.com/crawlers.json), which has every row with its source link. The dataset is under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/): reuse it, with credit and a link back.

- 2026-11-22 Added "Other documented agents": OAI-AdsBot (OpenAI) and Google-CloudVertexBot (Google), found on the operators' own crawler pages. Listed for completeness; they are neither answer engines, user-triggered fetchers nor training crawlers, so the audit doesn't score them.

- 2026-11-22 Added ipRanges: the IP range files OpenAI, Google, Bing, Perplexity and Apple publish for their agents, so requests can be checked against them. Other operators publish no machine-readable ranges we could find.

- 2026-11-22 Added Google-Agent and Google-GeminiNotebook, documented on Google's user-triggered fetchers page (it replaces Google-NotebookLM, listed as supported until August 2026). Google says these fetchers generally ignore robots.txt.

- 2026-11-21 Added Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-Training and Meta-ExternalFetcher after re-reading each operator's documentation.

- 2026-11-21 Amazonbot moved from search to training: Amazon says content it collects may be used to train Amazon's AI models.

- 2026-11-21 Google-Extended: noted that Google says it also controls grounding in Gemini Apps, not only training.

- 2026-11-21 Applebot: modelled Apple's rule that it follows the Googlebot group when Applebot is not named.

### Test your robots.txt

Paste your robots.txt and a path. Runs in your browser with the same parser as our audit (RFC 9309 longest-match, Apple's Googlebot fallback); nothing is sent anywhere.

| Agent | Kind | Result | Deciding rule |
| --- | --- | --- | --- |

### Build your robots.txt

Choose a stance for each kind of agent and each training crawler. Paste any Disallow lines you already use (admin, cart, account); the builder repeats them in every named group, because a crawler that matches a named group ignores User-agent: *.

Training crawlers and tokens (ticked = allowed) ·

```

```

### Copy-paste robots.txt

Be cited by AI answers and let future models learn your facts:

```
# Copy any Disallow lines from your "User-agent: *" group into each group below:
# a crawler that matches a named group ignores the * group.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Allow: /

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Google-Agent
User-agent: Google-GeminiNotebook
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
User-agent: MistralAI-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
User-agent: MistralAI-Training
Allow: /
```

Be cited, but opt out of training:

```
# Copy any Disallow lines from your "User-agent: *" group into each group below:
# a crawler that matches a named group ignores the * group.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Allow: /

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Google-Agent
User-agent: Google-GeminiNotebook
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
User-agent: MistralAI-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
User-agent: MistralAI-Training
Disallow: /
```

The [free audit](https://earncitations.com/#audit) checks your live robots.txt against all 26, and the [fix kit](https://earncitations.com/developers.html#fix) drafts these groups for you.

### Coming: one training signal for everyone

Google says it intends to support AIPREF, a shared machine-readable training-preference signal (from the IETF's AI Preferences working group), written as user-agent: __usage_ai-training, in early 2027. Until operators adopt it, per-bot groups are the only control that works.

### Questions

#### Does blocking GPTBot remove me from ChatGPT search?

No. OpenAI documents its agents as independent: OAI-SearchBot decides whether you appear in ChatGPT search results, GPTBot only governs training. You can allow one and block the other.

#### Does blocking Google-Extended affect Google Search?

Google says it does not affect inclusion or ranking in Google Search. But it is not only a training switch: Google says it also governs whether your content grounds answers in Gemini Apps and Vertex AI, so blocking it can cost you Gemini citations.

#### Do user-triggered fetchers obey robots.txt?

It varies by operator. Perplexity says Perplexity-User generally ignores robots.txt because a user requested the fetch; Amazon says Amzn-User may not follow all directives; Meta says Meta-ExternalFetcher may bypass robots.txt. Anthropic says Claude-User lets site owners control which pages it can fetch.

#### What happens if my robots.txt doesn't name Applebot?

Apple says Applebot follows your Googlebot rules if Applebot isn't named but Googlebot is. Otherwise the * group applies, as for every crawler.

#### What do "Google-Extended" and "Applebot-Extended" crawl?

Nothing. They are robots.txt tokens, not crawlers: Google and Apple crawl with their normal bots and read these groups to decide whether the content may be used for AI training (and, for Google-Extended, Gemini grounding).

#### How current is this list?

Each row was checked against the operator's own documentation on 2026-11-21. Operators add agents often: Amazon, Mistral and Meta all have agents that older lists may not include.
