Meta-ExternalAgent
Meta · Crawls public web content for training foundation AI models or improving products by indexing content directly
By Vinespire Editorial Team, Editorial ·
Update: Verified against Meta operator documentation for Meta-ExternalAgent.
This page is the operator/profile guide for Meta-ExternalAgent. To paste a full robots.txt and check Meta-ExternalAgent plus other AI crawlers in one pass, use the Robots.txt AI Crawler Validator. Short definitional entries also live in the AI search glossary.
Related free utilities are in free AI SEO tools.
What is Meta-ExternalAgent?
Meta-ExternalAgent is a Meta web crawler documented on Meta’s official web crawlers page for developers and webmasters. Meta states that this agent crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. In access logs, the User-Agent commonly appears similar to meta-externalagent/1.1, sometimes with a documentation URL fragment; robots.txt groups typically target the meta-externalagent product token. Meta documents that site managers can express crawl preferences with standard robots.txt Allow and Disallow rules for this crawler, and notes that robots.txt changes may take up to about 24 hours to take effect because crawlers may cache the file. Meta-ExternalAgent is distinct from Meta-WebIndexer (Meta AI search quality), Meta-ExternalFetcher (user-requested and agentic fetches that may bypass robots.txt), Meta-ExternalAds (advertising-related crawling), and facebookexternalhit (link previews when content is shared on Meta apps). Treat each Meta user-agent as a separate control surface. robots.txt remains a preference signal, not authentication: private or paywalled content still needs proper access control. Always re-check Meta’s current web crawlers documentation when adjusting policy, because operators add and refine agents over time.
How to block Meta-ExternalAgent
Add this group to robots.txt to disallow meta-externalagent. Path rules can be narrowed if you only need to protect parts of the site.
User-agent: meta-externalagent Disallow: /
How to allow Meta-ExternalAgent
For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.
User-agent: meta-externalagent Allow: /
Should you block Meta-ExternalAgent?
Blocking Meta-ExternalAgent is a common choice when you want to limit Meta’s documented training-oriented or direct-indexing crawl of public pages. Allowing it is reasonable if open participation in Meta’s AI and product improvement pipelines matches your licensing and brand goals. Blocking this agent does not by itself stop Meta-WebIndexer, Meta-ExternalFetcher, Meta-ExternalAds, or facebookexternalhit—configure each token explicitly if you need a full policy. GEO-oriented teams often distinguish training crawl (more frequently restricted) from search-indexing agents that affect citations in AI answers; Meta’s docs describe Meta-WebIndexer separately for Meta AI search quality. After any robots.txt change, allow time for cache refresh and confirm behavior in logs. Do not rely on robots.txt alone for non-public URLs. Document the decision so legal, SEO, and security stakeholders share the same intent, and revisit when Meta updates its crawler list.
Category: Training
Last verified: . Re-check operator docs after major crawler announcements.
Related bots (Training)
Meta-ExternalAgent FAQ
- Meta documents robots.txt controls for Meta-ExternalAgent and shows Allow/Disallow examples. Changes may take up to about 24 hours because crawlers can cache robots.txt. Confirm on Meta’s web crawlers documentation.