Does Googlebot respect robots.txt?

Yes. Google documents that common crawlers including Googlebot always obey robots.txt rules for automatic crawls.

Is Googlebot the same as Google-Extended?

No. Googlebot supports Search indexing and related Search features. Google-Extended is a separate product token for Gemini training/grounding preferences and does not impact Search inclusion the same way.

Will blocking Googlebot improve AI training opt-out?

No. For Gemini-related training/grounding preferences, Google documents Google-Extended. Blocking Googlebot primarily harms Search discovery.

How do I verify real Googlebot traffic?

Use Google’s official reverse DNS and IP range verification guidance. User-Agent strings alone can be spoofed.

What products does Googlebot affect?

Google documents impacts on Google Search (including Discover and Search features) and related products depending on specialized Googlebot variants.

Should public sites allow Googlebot?

Nearly all sites that want Google Search visibility allow Googlebot on indexable public content and use noindex or auth for pages that should stay out.

Googlebot

Google · Primary Google crawler family for Google Search indexes and related Search surfaces (not Google-Extended generative preferences)

By Vinespire Editorial Team, Editorial · Last reviewed July 19, 2026

Update: Verified against Google operator documentation for Googlebot.

See our sourcing methodology →

This page is the operator/profile guide for Googlebot. To paste a full robots.txt and check Googlebot plus other AI crawlers in one pass, use the Robots.txt AI Crawler Validator. Short definitional entries also live in the AI search glossary.

Related free utilities are in free AI SEO tools.

What is Googlebot?

Googlebot is Google’s primary common crawler for building Search indexes and related products. Official documentation describes smartphone and desktop HTTP user-agent patterns containing Googlebot/2.1, robots.txt token Googlebot, and impacts on Google Search (including Discover and Search features) plus related surfaces such as Images, Video, and News depending on specialized variants. Google’s common crawlers always obey robots.txt for automatic crawls. Googlebot is not Google-Extended: Google-Extended is a standalone product token controlling whether crawled content may be used for training future Gemini models and certain grounding uses, without impacting Search inclusion as a ranking signal per Google’s crawler docs. Site owners optimizing for classic SEO almost always allow Googlebot on public indexable content. Specialized tokens (Googlebot-Image, Googlebot-Video, Googlebot-News, Storebot-Google, Google-InspectionTool, and others) exist for narrower product surfaces. For AI visibility programs, keep Googlebot Search access distinct from Google-Extended AI preferences and from Google-CloudVertexBot enterprise agent crawls. Verify requests via Google’s published reverse DNS and IP range guidance rather than User-Agent alone.

How to block Googlebot

Add this group to robots.txt to disallow Googlebot. Path rules can be narrowed if you only need to protect parts of the site.

User-agent: Googlebot
Disallow: /

How to allow Googlebot

For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.

User-agent: Googlebot
Allow: /

Should you block Googlebot?

Blocking Googlebot removes or severely limits Google Search crawl of affected URLs and is rarely appropriate for public marketing content. Prefer noindex, authentication, or path-level Disallow only where you truly do not want Search discovery. AI training opt-out for Gemini-related uses is handled with Google-Extended, not by blocking Googlebot—Google documents that Google-Extended does not impact Search inclusion the same way. If WAF rules challenge Googlebot, Search indexing and rich-result tooling can break; allowlist verified Google crawler ranges. Do not assume blocking Googlebot stops GoogleOther, Storebot-Google, or CloudVertexBot. Re-read Google’s crawler overview when diagnosing coverage drops.

Category: Search indexing

Official documentation (Google)

Last verified: 2026-07-19. Re-check operator docs after major crawler announcements.

Googlebot FAQ

Yes. Google documents that common crawlers including Googlebot always obey robots.txt rules for automatic crawls.