AI Crawler & Bot Directory
AI crawlers are automated agents that request public web pages for model training, answer-engine search indexes, structured datasets, or live assistant browsing. They use distinct User-agent names in robots.txt—different from classic search crawlers like Googlebot—and each major AI company may run several bots for different purposes. This directory compares them side by side so you can decide what to allow or block with clear, verifiable facts.
By Vinespire Editorial Team, Editorial · Reviewed by Vinespire Editorial Team, Editorial ·
Update: Registry verified against operator docs. Hub stamp 2026-07-19; each bot page carries its own lastVerified date.
All AI bots
Filter by category or sort by operator. Every row is present in the initial HTML for crawlers; filters only hide rows in the browser.
Showing 29 of 29
| Bot name | Operator | Purpose | Respects robots.txt? | Category |
|---|---|---|---|---|
| AI2Bot AI2Bot | Allen Institute for AI (Ai2) | Explores certain domains for web content used to train open language models at the Allen Institute for AI | Partial | Training |
| Amazonbot Amazonbot | Amazon | Amazon’s crawler for improving products such as Alexa and AI experiences | Yes | Training |
| Claude-SearchBot Claude-SearchBot | Anthropic | Indexes content to improve Claude search result relevance | Yes | Search indexing |
| Claude-User Claude-User | Anthropic | Fetches pages when people use Claude and need live web content | Yes | Live browsing |
| ClaudeBot ClaudeBot | Anthropic | Collects web content that may contribute to Claude model training and safety | Yes | Training |
| Applebot Applebot | Apple | Apple’s web crawler for Spotlight, Siri, Safari search features; crawled data may also support generative features unless opted out via Applebot-Extended | Yes | Search indexing |
| Applebot-Extended Applebot-Extended | Apple | Preferences for Apple Intelligence / generative features (related to Applebot) | Yes | Dataset / other |
| Bytespider Bytespider | ByteDance | ByteDance web crawler associated with AI / content products | Yes | Training |
| cohere-ai cohere-ai | Cohere | Cohere crawler / bot identifier used in robots.txt for AI data preferences | Yes | Training |
| CCBot CCBot | Common Crawl | Builds the Common Crawl open web dataset used by many AI researchers and companies | Yes | Dataset / other |
| Diffbot Diffbot | Diffbot | Extracts structured data from the web for knowledge graphs and AI products | Yes | Dataset / other |
| DuckAssistBot DuckAssistBot | DuckDuckGo | Real-time crawler for DuckDuckGo AI-assisted answers that cite sources (not used to train AI models per DuckDuckGo) | Yes | Search indexing |
| DuckDuckBot DuckDuckBot | DuckDuckGo | Primary DuckDuckGo search crawler for improving organic search results and the private search experience | Yes | Search indexing |
| Google-CloudVertexBot Google-CloudVertexBot | Crawls sites when site owners request website data for building Vertex AI Agents; does not affect Google Search rankings | Yes | Dataset / other | |
| Google-Extended Google-Extended | Controls use of content for Gemini and Google AI features (not classic Googlebot search) | Yes | Dataset / other | |
| Googlebot Googlebot | Primary Google crawler family for Google Search indexes and related Search surfaces (not Google-Extended generative preferences) | Yes | Search indexing | |
| GoogleOther GoogleOther | Generic Google crawler for product teams fetching publicly accessible content (for example internal R&D); not tied to a single consumer product | Yes | Dataset / other | |
| Storebot-Google Storebot-Google | Google crawler for Google Shopping surfaces (Shopping tab, Google Shopping) with dedicated robots.txt token Storebot-Google | Yes | Dataset / other | |
| FacebookExternalHit facebookexternalhit | Meta | Fetches shared links to build previews (title, description, image) on Facebook, Instagram, Messenger, and related Meta apps | Partial | Dataset / other |
| Meta-ExternalAds meta-externalads | Meta | Crawls the web for advertising and other business-related product and service improvements | Yes | Dataset / other |
| Meta-ExternalAgent meta-externalagent | Meta | Crawls public web content for training foundation AI models or improving products by indexing content directly | Yes | Training |
| Meta-ExternalFetcher meta-externalfetcher | Meta | Fetches individual links at a user’s request and supports agentic AI product functions that may navigate websites | Partial | Live browsing |
| Meta-WebIndexer meta-webindexer | Meta | Crawls the web to improve Meta AI search result quality and support citing and linking sources in responses | Yes | Search indexing |
| ChatGPT-User ChatGPT-User | OpenAI | Fetches pages when users browse the web during ChatGPT conversations | Partial | Live browsing |
| GPTBot GPTBot | OpenAI | Crawls public web content that may support OpenAI models and products | Yes | Training |
| OAI-AdsBot OAI-AdsBot | OpenAI | Validates safety and relevance of landing pages submitted as ads on ChatGPT; not used to train foundation models | Yes | Dataset / other |
| OAI-SearchBot OAI-SearchBot | OpenAI | Indexes content for ChatGPT search / retrieval features | Yes | Search indexing |
| Perplexity-User Perplexity-User | Perplexity | User-triggered fetches when Perplexity retrieves a page in-session | Partial | Live browsing |
| PerplexityBot PerplexityBot | Perplexity | Crawls and indexes pages for Perplexity search and answers | Yes | Search indexing |
All bots remain in the page HTML for crawlers; filters only hide rows visually.
Which bots should I block or allow?
There is no universal “block all AI” or “allow everything” answer that fits every site. The useful approach is to match each bot’s category to your goals: AI visibility and citations, content licensing control, crawl-budget hygiene, or a mix of all three. Vinespire’s default editorial stance is pro-visibility for public marketing and documentation content, while staying factually neutral about the mechanics of each agent.
Training crawlers (for example GPTBot, ClaudeBot, Amazonbot, Bytespider, cohere-ai) collect public pages that may support model development. Allowing them can help future models know accurate facts about your brand; blocking them is appropriate when licensing, paywalls, or policy forbid training reuse. Remember that blocking one company’s training bot does not block another’s—and it does not remove content already present in older datasets.
Search-indexing bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) influence whether pages can appear in AI product search layers. Brands investing in generative engine optimization usually allow these agents and invest in clear, crawlable pages. A catch-all User-agent: * Disallow: / often blocks them by accident.
Live-browsing agents (ChatGPT-User, Claude-User, Perplexity-User) typically fetch pages when a person asks an assistant to open or use a URL. Volume is usually lower and demand-driven. Many sites allow them so conversational tools show accurate content; some operators document partial robots.txt applicability for user-fetch agents—read each bot’s official docs.
Dataset and product-token crawlers such as CCBot (Common Crawl), Diffbot, Google-Extended, and Applebot-Extended sit slightly outside the “one chat brand” mental model. CCBot feeds open corpora used by many downstream projects; Google-Extended and Applebot-Extended express preferences for generative product uses without necessarily equating to classic search crawlers. Knowing they exist matters even when they are less famous than GPTBot.
Practical workflow: inventory which agents you care about, write explicit robots.txt groups (avoid relying only on wildcards), deploy, then validate. robots.txt is not authentication—protect private content with real access controls.
Robots.txt AI Validator
Paste robots.txt and see which AI crawlers are allowed or blocked.
llms.txt Generator
Publish a curated site summary language models can retrieve at /llms.txt.
Free tools hub
More GEO utilities: readiness checks, schema generators, and estimators.
Glossary: AI Crawler · User-Agent · Training Data (glossary routes ship as the glossary hub expands).
Frequently asked questions
- An AI crawler (or AI bot) is software that requests public web pages on behalf of an AI company or data project—often to train models, build search indexes for answer engines, extract structured data, or fetch a page when a user asks an assistant to open a URL. They are related to traditional search crawlers but serve different product goals and use distinct User-agent names in robots.txt.