MetaMeta Is Crawling the Web Like It's Building a Search Index

In mid-July 2026, Meta-WebIndexer, Meta's web-indexing crawler, accounted for roughly 2.2% of all AI crawler requests in our crawler logs. By August 9, that share had climbed to 37.8%. That is more than a third of all AI crawler traffic we track, and a roughly 17x increase in under a month, from a single crawler. The surge lines up with reports that Meta is building its own web search index so its AI doesn't have to rely on Google or Bing. This report tracks Meta-WebIndexer's share day by day.

Klaas Foppen](/content/blog/author/klaas/index.html)·August 10, 2026

MetaMeta Is Building Its Own Search Engine for AI

Meta-WebIndexer's daily share of the AI crawler requests Promptwatch tracks (listed below the chart). This is not total site traffic, and it excludes Meta's other crawlers. The share hovered around 2% until July 17, spiked to ~23% between July 20 and 22, then surged from August 5 to a peak of 37.8% on August 9. August 10 is a partial day.

% of requests

What this means for you

A month ago, Meta-WebIndexer was a minor player in AI crawling, about 1 in 50 AI crawler requests we tracked. By August 9, more than 1 in 3 AI crawler requests came from this single crawler, which makes it the heaviest AI crawler in our logs. The surge comes from Meta's indexing crawler specifically, not its AI-training or link-fetching bots. Crawling at this scale and pace is what building a web index from scratch looks like.

Google could learn from every search Meta AI sends its way, and reports say Meta wants its own index precisely to cut that dependency. If Meta ships its own search engine for Meta AI, whether your content was crawled during this build-out determines whether you can be cited in it.

How to act on it

• Check your own server logs and robots.txt for Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher, and FacebookBot. If Meta is building a search index, blocking these crawlers today decides whether your content is in it at launch.
• Treat Meta as an emerging AI search surface rather than only a social platform. Meta-WebIndexer exists to improve Meta AI search results and cite sources, the same citation dynamic you already optimize for with ChatGPT and Perplexity.
• Watch crawl load. Meta's crawl volume grew fast enough to trigger load alerts on independent sites. If your infrastructure bills by request, budget for sustained Meta crawler traffic rather than treating it as a spike.

The tweet that put a spotlight on Meta's crawling

On August 6, right as Meta-WebIndexer's share was climbing past 26% in our logs, @levelsio posted that Meta staff had told him the company is allegedly building its own web index, and that Meta was scraping his sites heavily enough to trigger server load alerts.

@levelsio

@levelsio

· Follow

View on Twitter

Meta staff DM'd me secretly

Posted with permission

Meta is ALLEGEDLY building their own Google search engine, so that if their AI does a web search it doesn't end up at Google, as Google could then use it for THEIR training, so they want their own web index that they will then Show more

@levelsio

@levelsio

Meta is doing heavy heavy heavy scraping on all my sites this week too (and seemingly everyone else's sites now)

So much so that I got load average alerts for it today on one VPS

The fact that they're hitting url2og (my own screenshot service for all my sites) might they're

Which crawlers count toward this percentage

The share above is Meta-WebIndexer requests (the meta-webindexer user agent only) divided by requests from all AI crawlers Promptwatch tracks, identified by user agent. Meta's other crawlers below count toward the denominator, but not toward Meta-WebIndexer's share. Regular search engine crawlers (Googlebot, Bingbot), link-preview fetchers like FacebookExternalHit, and human visitors are excluded entirely.

• OpenAI GPTBot OAI-SearchBot ChatGPT-User OAI-AdsBot
• Anthropic ClaudeBot Claude-User Claude-SearchBot claude-web
• Google Google-Extended Google-Agent
• Perplexity PerplexityBot Perplexity-User
• xAI GrokBot xAI-Grok Grok-DeepSearch
• Mistral MistralAI-User MistralAI-Index
• Meta meta-webindexer meta-externalagent meta-externalfetcher FacebookBot
• Cohere cohere-ai
• DeepSeek DeepSeekBot

How we collect this data

We collect millions of prompt responses, citations, and click data from the actual user interfaces of major AI platforms: over 26 billion data points and growing. This gives us one of the largest datasets on how AI search engines cite sources and recommend brands.

Real UI monitoring

Data straight from the interfaces of ChatGPT, Gemini, Perplexity, Claude, AI Overviews, and more.

26B+ data points

Over 26 billion analyzed citations, prompts, and responses, one of the largest AI search datasets available.

Continuously updated

Refreshed constantly so the trends you see reflect the latest behavior of AI search engines.

Aggregated & public

Published freely for the GEO community, based on aggregated, non-identifiable trends.

Track AI Crawlers on Your Site

Meta-WebIndexer went from 2% to over a third of all tracked AI crawler traffic in under a month. Track which AI crawlers hit your site and how AI search engines cite you, so shifts like this don't catch you off guard.