Does GPTBot respect robots.txt?

OpenAI documents that GPTBot honors robots.txt. Use an explicit User-agent: GPTBot group with Allow or Disallow rules. Verify details on OpenAI’s official bot documentation and re-check after policy updates.

Is GPTBot the same as ChatGPT-User?

No. GPTBot is OpenAI’s broader crawl/training-oriented agent. ChatGPT-User is used for live fetches during ChatGPT conversations. OAI-SearchBot is a third token for search indexing. Configure each separately.

Does blocking GPTBot affect my Google rankings?

No. Google Search uses Googlebot (and related Google crawlers), not GPTBot. Blocking GPTBot only affects OpenAI’s documented use of that agent. It does not replace noindex or Google Search Console settings.

What User-agent string does GPTBot use?

The robots.txt token is GPTBot. Log lines may include additional version or URL details depending on the client, but robots.txt matching uses the GPTBot product token as documented by OpenAI.

Should documentation sites allow GPTBot?

Many public docs sites allow GPTBot so models can learn accurate product facts. Teams with strict licensing may block it. Review legal requirements, then test your robots.txt with Vinespire’s validator.

How do I allow only GPTBot but block other AI bots?

Write a specific Allow group for GPTBot and Disallow groups for other agents you want closed. Avoid a bare User-agent: * Disallow: / unless you also add explicit Allow overrides for every bot you want open.

GPTBot

OpenAI · Crawls public web content that may support OpenAI models and products

What is GPTBot?

GPTBot is OpenAI’s primary web crawler for collecting publicly available content that may be used to improve models and related products. Site owners typically see it in server logs as the User-agent token GPTBot. OpenAI documents how to control GPTBot with robots.txt and publishes IP ranges and bot details for operators who want to allow or restrict access. Blocking GPTBot can reduce how OpenAI systems learn from your public pages over time; allowing it is a common choice for brands that want broader presence in AI-related systems. GPTBot is distinct from ChatGPT-User (live conversation fetches) and OAI-SearchBot (search indexing). Treat each token as a separate control surface in robots.txt.

How to block GPTBot

Add this group to robots.txt to disallow GPTBot. Path rules can be narrowed if you only need to protect parts of the site.

User-agent: GPTBot
Disallow: /

How to allow GPTBot

For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.

User-agent: GPTBot
Allow: /

Should you block GPTBot?

Allowing GPTBot generally aligns with AI visibility goals if your public content is something you want models to know about. Blocking is reasonable when you have licensing constraints, paywalled material, or a deliberate content-control policy. Blocking GPTBot does not block ChatGPT-User or OAI-SearchBot—configure each agent separately. robots.txt is not a security boundary; authenticated or non-public URLs should still use proper access control. For most marketing and documentation sites, allowing GPTBot and monitoring logs is a balanced default; review after major product or policy changes from OpenAI.

Category: Training

Official documentation (OpenAI)

Last verified: 2026-07-18. Re-check operator docs after major crawler announcements.

GPTBot FAQ

OpenAI documents that GPTBot honors robots.txt. Use an explicit User-agent: GPTBot group with Allow or Disallow rules. Verify details on OpenAI’s official bot documentation and re-check after policy updates.