AI Crawler
Definition
An AI crawler is a bot that fetches public web pages for an AI company or dataset—often for training, indexing, or live assistant browsing.
By Vinespire Editorial Team, Editorial ·
Crawlers & botsAI search glossary
This term is part of the full AI search glossary.
Full definition
Examples include GPTBot, ClaudeBot, PerplexityBot, and CCBot. They differ from classic Googlebot in purpose even when robots.txt syntax looks similar. Operators may split training, search, and user-fetch agents into separate tokens.
Site owners decide allow/block policy per agent. See robots.txt and the AI Bot Directory concepts under GPTBot and ClaudeBot.