User-Agent
Definition
A User-Agent is an identifier string a client sends to a server to describe the browser, bot, or application making the request.
By Vinespire Editorial Team, Editorial ·
Crawlers & botsAI search glossary
This term is part of the full AI search glossary.
Full definition
In robots.txt, User-agent: lines select which bot a rule group applies to (for example GPTBot or *). In HTTP logs, the User-Agent header helps operators distinguish crawlers from browsers and measure AI-related traffic separately from classic searchbots.
Spoofing is possible; User-Agent alone is not authentication. Operators who need stronger verification may combine published IP ranges, reverse DNS patterns, and WAF bot scores where vendors provide them.
AI companies often run multiple tokens for training, search indexing, and live browsing—configure each explicitly. See robots.txt and AI Crawler.