Meta-WebIndexer
Meta · Crawls the web to improve Meta AI search result quality and support citing and linking sources in responses
By Vinespire Editorial Team, Editorial ·
Update: Verified against Meta operator documentation for Meta-WebIndexer.
This page is the operator/profile guide for Meta-WebIndexer. To paste a full robots.txt and check Meta-WebIndexer plus other AI crawlers in one pass, use the Robots.txt AI Crawler Validator. Short definitional entries also live in the AI search glossary.
Related free utilities are in free AI SEO tools.
What is Meta-WebIndexer?
Meta-WebIndexer is documented by Meta as a crawler that navigates the web to improve Meta AI search result quality for users. Meta states that it analyzes online content to enhance relevance and accuracy of Meta AI, and that allowing Meta-WebIndexer in robots.txt helps Meta cite and link to your content in Meta AI responses. Log User-Agent strings are described as similar to meta-webindexer/1.1 (with or without a documentation reference). This agent is a search-quality / AI answer surface control, not the same product as Meta-ExternalAgent (training or direct indexing uses) or Meta-ExternalFetcher (on-demand and agentic fetches). Site owners should use an explicit User-agent: meta-webindexer group rather than assuming a single Meta block covers every crawler. Meta’s webmaster guidance emphasizes industry-standard robots.txt over non-standard tags, with cached robots.txt updates potentially delayed up to about 24 hours. For GEO and AEO planning, Meta-WebIndexer is the closer analogue to other answer-engine search crawlers: participation can affect whether your public pages are available as citable material in Meta AI search contexts, subject to Meta’s systems and eligibility. Verify spelling and purpose against Meta’s current web crawlers documentation before publishing policy.
How to block Meta-WebIndexer
Add this group to robots.txt to disallow meta-webindexer. Path rules can be narrowed if you only need to protect parts of the site.
User-agent: meta-webindexer Disallow: /
How to allow Meta-WebIndexer
For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.
User-agent: meta-webindexer Allow: /
Should you block Meta-WebIndexer?
If you want public content eligible for Meta AI search contexts and potential citations, allowing Meta-WebIndexer aligns with Meta’s documented purpose for this agent. Blocking is appropriate when you prefer not to participate in that retrieval/indexing path or when crawl cost outweighs the benefit. Blocking Meta-WebIndexer does not automatically enforce a training opt-out for Meta-ExternalAgent, and it does not stop user-triggered Meta-ExternalFetcher behavior that Meta says may bypass robots.txt. Brands optimizing for AI visibility often allow search-oriented agents while setting a separate policy on training crawlers—only if that split matches legal requirements. After changing robots.txt, monitor logs for the meta-webindexer token and re-read Meta’s official page if new agents appear. Pair crawl access with clear, accurate public content if inclusion is the goal.
Category: Search indexing
Last verified: . Re-check operator docs after major crawler announcements.
Related bots (Search indexing)
Meta-WebIndexer FAQ
- Meta presents Meta-WebIndexer with standard robots.txt preference controls alongside its other listed crawlers. Use an explicit User-agent: meta-webindexer group.