Home / AI crawler directory
Directory checked 2026-11-21 against each operator's docs

AI crawler directory

26 agents from 12 operators, in three kinds: answer engines that index you, fetchers that read you live when a user asks, and training crawlers. Each description paraphrases the operator's own documentation, linked in the last column. "Named by" is how many of the 119 software sites in our research scan name the agent in robots.txt; agents added after that scan show a dash.

Index pages that AI answers cite. Block one and your pages can drop out of the AI search it feeds.

AgentOperatorWhat it does (per the operator)Named bySource
OAI-SearchBotOpenAISurfaces websites in search results in ChatGPT's search features.24docs
Claude-SearchBotAnthropicNavigates the web to improve search result quality for Claude users. Blocking it may reduce your visibility in Claude's answers.11docs
PerplexityBotPerplexitySurfaces and links websites in Perplexity search results.27docs
GooglebotGoogleGoogle Search's main crawler.6docs
BingbotMicrosoftBing's standard crawler; handles most of Bing's crawling.11docs
ApplebotApplePowers search in Spotlight, Siri and Safari; its data may also ground AI answers in Siri and Search. Follows your Googlebot rules if Applebot isn't named.2docs
Amzn-SearchBotAmazonMakes content eligible for search experiences such as Alexa. Not used for generative AI training.—docs
DuckAssistBotDuckDuckGoCrawls pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources.8docs
MistralAI-IndexMistralIndexes content for Mistral search, used to answer questions in Vibe. Not used for training.—docs

User-triggered fetchers

Load a page live when someone asks an assistant about it. Not crawlers in the usual sense; several operators say they may not follow robots.txt.

AgentOperatorWhat it does (per the operator)Named bySource
ChatGPT-UserOpenAIVisits a page when a ChatGPT or Custom GPT user's request needs it. Not used for automatic crawling or to decide what appears in search.26docs
Claude-UserAnthropicFetches pages when someone asks Claude a question. Blocking it stops Claude retrieving your content for those users.12docs
Perplexity-UserPerplexityFetches pages for user actions in Perplexity. Perplexity says it generally ignores robots.txt because a user requested the fetch.11docs
Google-AgentGoogleUsed by agents hosted on Google infrastructure to navigate the web and perform actions at a user's request. Google says its user-triggered fetchers generally ignore robots.txt.—docs
Google-GeminiNotebookGoogleFetches individual URLs that Gemini Notebook users add as sources (it replaces Google-NotebookLM, which Google lists as supported until August 2026). Google says its user-triggered fetchers generally ignore robots.txt.—docs
Meta-ExternalFetcherMetaFetches individual links at a user's request, including for AI agents completing tasks. Meta says it may bypass robots.txt.—docs
Amzn-UserAmazonFetches live information for user requests such as Alexa questions. May not follow all robots.txt directives. Not used for training.—docs
MistralAI-UserMistralFetches pages for user actions in Vibe.3docs

Training crawlers and tokens

Collect content for model training. Blocking them is a legitimate choice; future models will just know less about you.

AgentOperatorWhat it does (per the operator)Named bySource
GPTBotOpenAICrawls content that may be used to train OpenAI's generative AI foundation models. Independent of OAI-SearchBot: blocking it doesn't remove you from ChatGPT search.29docs
ClaudeBotAnthropicCollects web content that could contribute to Anthropic's model training. Blocking it excludes your future content from training datasets.26docs
Google-Extended (token)GoogleA robots.txt token, not a crawler. Controls whether content Google crawls may train future Gemini models and ground answers in Gemini Apps and Vertex AI. No effect on Google Search.24docs
Applebot-Extended (token)AppleA robots.txt token, not a crawler. Disallow it to opt out of training Apple's foundation models; Applebot still crawls for search.16docs
CCBotCommon CrawlBuilds Common Crawl's open web archive, which many AI training datasets draw on.21docs
Meta-ExternalAgentMetaCrawls for use cases such as training foundation AI models or improving products (Meta's wording).15docs
BytespiderByteDanceByteDance's crawler. We found no official documentation of its purpose or robots.txt behaviour; classed as training based on public reporting.14none found
AmazonbotAmazonUsed to improve Amazon products and services; Amazon says it may be used to train Amazon AI models.12docs
MistralAI-TrainingMistralCrawls web content to build datasets for training Mistral models.—docs

Other documented agents

Operators document these next to their AI crawlers, but they don't index you for answers, fetch for a user, or train models, so the audit doesn't score them. If you name them in robots.txt, the usual rules apply.

AgentOperatorWhat it doesSource
OAI-AdsBotOpenAIVisits landing pages submitted as ads on ChatGPT to check them against OpenAI's policies, and may use their content to decide when to show the ad. OpenAI says it only visits pages submitted as ads and its data isn't used to train foundation models. IP ranges.developers.openai.com
Google-CloudVertexBotGoogleCrawls a site when its owner asks for it while building Vertex AI Agents. Google says the token has no effect on Google Search or other products.developers.google.com

Changes

Operators add and repurpose agents often. Every change to this list is dated here and in the open dataset (crawlers.json), which has every row with its source link. The dataset is under CC BY 4.0: reuse it, with credit and a link back.

Test your robots.txt

Paste your robots.txt and a path. Runs in your browser with the same parser as our audit (RFC 9309 longest-match, Apple's Googlebot fallback); nothing is sent anywhere.

AgentKindResultDeciding rule

Build your robots.txt

Choose a stance for each kind of agent and each training crawler. Paste any Disallow lines you already use (admin, cart, account); the builder repeats them in every named group, because a crawler that matches a named group ignores User-agent: *.

Training crawlers and tokens (ticked = allowed) ·

Copy-paste robots.txt

Be cited by AI answers and let future models learn your facts:

# Copy any Disallow lines from your "User-agent: *" group into each group below:
# a crawler that matches a named group ignores the * group.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Allow: /

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Google-Agent
User-agent: Google-GeminiNotebook
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
User-agent: MistralAI-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
User-agent: MistralAI-Training
Allow: /

Be cited, but opt out of training:

# Copy any Disallow lines from your "User-agent: *" group into each group below:
# a crawler that matches a named group ignores the * group.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Allow: /

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Google-Agent
User-agent: Google-GeminiNotebook
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
User-agent: MistralAI-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
User-agent: MistralAI-Training
Disallow: /

The free audit checks your live robots.txt against all 26, and the fix kit drafts these groups for you.

Coming: one training signal for everyone

Google says it intends to support AIPREF, a shared machine-readable training-preference signal (from the IETF's AI Preferences working group), written as user-agent: __usage_ai-training, in early 2027. Until operators adopt it, per-bot groups are the only control that works.

Questions

Does blocking GPTBot remove me from ChatGPT search?

No. OpenAI documents its agents as independent: OAI-SearchBot decides whether you appear in ChatGPT search results, GPTBot only governs training. You can allow one and block the other.

Does blocking Google-Extended affect Google Search?

Google says it does not affect inclusion or ranking in Google Search. But it is not only a training switch: Google says it also governs whether your content grounds answers in Gemini Apps and Vertex AI, so blocking it can cost you Gemini citations.

Do user-triggered fetchers obey robots.txt?

It varies by operator. Perplexity says Perplexity-User generally ignores robots.txt because a user requested the fetch; Amazon says Amzn-User may not follow all directives; Meta says Meta-ExternalFetcher may bypass robots.txt. Anthropic says Claude-User lets site owners control which pages it can fetch.

What happens if my robots.txt doesn't name Applebot?

Apple says Applebot follows your Googlebot rules if Applebot isn't named but Googlebot is. Otherwise the * group applies, as for every crawler.

What do "Google-Extended" and "Applebot-Extended" crawl?

Nothing. They are robots.txt tokens, not crawlers: Google and Apple crawl with their normal bots and read these groups to decide whether the content may be used for AI training (and, for Google-Extended, Gemini grounding).

How current is this list?

Each row was checked against the operator's own documentation on 2026-11-21. Operators add agents often: Amazon, Mistral and Meta all have agents that older lists may not include.