Meta-ExternalFetcher
Meta · Fetches individual links at a user’s request and supports agentic AI product functions that may navigate websites
By Vinespire Editorial Team, Editorial ·
Update: Verified against Meta operator documentation for Meta-ExternalFetcher.
This page is the operator/profile guide for Meta-ExternalFetcher. To paste a full robots.txt and check Meta-ExternalFetcher plus other AI crawlers in one pass, use the Robots.txt AI Crawler Validator. Short definitional entries also live in the AI search glossary.
Related free utilities are in free AI SEO tools.
What is Meta-ExternalFetcher?
Meta-ExternalFetcher is documented by Meta as a crawler that fetches individual links at a user’s request and supports product functions such as evaluating and improving agentic AI capabilities—including helping AI navigate websites to complete tasks for users. Meta explicitly states that this crawler may bypass robots.txt rules because fetches are tied to user-requested or agentic workflows. Log strings are described as similar to meta-externalfetcher/1.1. That partial robots.txt applicability is the critical operational difference from Meta-ExternalAgent and Meta-WebIndexer, which Meta presents with standard robots.txt preference controls. Site owners should not assume a Disallow group fully stops Meta-ExternalFetcher; sensitive paths need authentication, network controls, or other hard restrictions. Category-wise it is a live-browsing / on-demand agent more than a bulk training crawler, though Meta’s product description also covers agentic navigation use cases. Keep policies for meta-externalfetcher separate from other Meta tokens, and re-verify Meta’s web crawlers documentation when designing WAF or bot-management exceptions. Contact paths for Meta webmaster questions are documented on the same official page.
How to block Meta-ExternalFetcher
Add this group to robots.txt to disallow meta-externalfetcher. Path rules can be narrowed if you only need to protect parts of the site.
User-agent: meta-externalfetcher Disallow: /
How to allow Meta-ExternalFetcher
For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.
User-agent: meta-externalfetcher Allow: /
Should you block Meta-ExternalFetcher?
robots.txt alone is an incomplete control for Meta-ExternalFetcher because Meta documents that this agent may bypass robots.txt for user-requested and agentic fetches. Publishing a Disallow group can still express preference and may affect some compliant paths, but you should not treat it as a guarantee. For public marketing and documentation sites, allowing on-demand fetches can help Meta AI product flows read the live page when a user or agent needs it. For private app areas, authenticated resources, or regulated content, rely on login walls, authorization, and infrastructure controls rather than robots.txt. If crawl volume or abuse is the concern, combine bot management, rate limits, and IP allow/deny lists with log review. Coordinate security and product teams so a blanket “block all Meta bots” policy does not create false confidence about this fetcher. Re-check official docs after Meta product changes.
Category: Live browsing
Last verified: . Re-check operator docs after major crawler announcements.
Related bots (Live browsing)
Meta-ExternalFetcher FAQ
- Meta documents that Meta-ExternalFetcher may bypass robots.txt because it performs user-requested and agentic fetches. Treat robots.txt as partial for this agent and protect sensitive paths with authentication.