AI Crawlers Scrape 11,122 Pages per Visit Returned — Most AI Visibility Metrics Measure the Wrong Layer
Anthropic's crawler scrapes 11,122 pages for every visit it sends back. Most AI visibility dashboards report crawler traffic, not citations. Here are the four metrics that actually tell you whether AI engines are sending buyers your way.
Anthropic's AI crawler scrapes 11,122 publisher pages for every single visit it sends back. OpenAI's ratio is 857 to 1. Google's traditional crawler? Five to one. If your AI visibility dashboard is counting bot traffic, you are measuring the scraping — not the citing. The metric most brands report to their boards has almost no relationship to whether AI engines actually recommend them.
The Crawl-to-Referral Ratio That Undermines the AI Visibility Pitch
Publishers are racing to sell AI visibility as the next advertising currency. Time, Axios, and Forbes are all packaging their AI citation prominence into advertiser pitches, positioning it as the successor to Comscore. The problem: the number underneath the pitch is built on crawler traffic, and crawler traffic is almost entirely one-directional.
Cloudflare data for the week of May 25–June 1, 2026, shows the gap between what AI crawlers take and what they return:
| AI Crawler | Pages Scraped per Referral Sent | Change From Prior Month |
|---|---|---|
| Anthropic (ClaudeBot) | 11,122 to 1 | Down from 13,528 to 1 |
| OpenAI (GPTBot) | 857 to 1 | Down from 1,252 to 1 |
| Perplexity | 190 to 1 | Up from 95 to 1 |
| Google (Googlebot) | 5 to 1 | Roughly steady |
Time's COO Mark Howard admitted the measurement gap openly: "They all have different methodologies. Their numbers are all different. I don't think that there's any standardization here that you can anchor on, at the moment." Time uses bot traffic as its main proxy with clients — daily AI crawler activity compared against human visits.
That proxy is the wrong layer. Bots crossed 57.5% of total HTML web traffic in June 2026 according to Cloudflare's CEO, but only 9.3% of AI crawler requests serve live search — the rest is training and indexing. The bots most likely to actually cite your content in a live answer are a small fraction of what hits your server logs.
62% of AI Citations Are Ghost Citations
Even when an AI engine does cite you, it probably will not mention your brand. Research from Kevin Indig across 3,981 domains and 115 prompts found that 62% of all AI citations are ghost citations — the AI includes a source link but never names the brand in its answer text.
The effective brand visibility rate drops from 53.1% to 10.6% once you separate citations with brand mentions from links buried in footnotes. You supply the facts. The AI takes the credit. Informational content — the guides, how-to articles, and research pages that most brands invest in — is the most vulnerable to this extraction pattern because AI engines can lift the facts without needing to reference the source.
I wrote earlier this week about how AI engines rewrite 76% of cited content before presenting it. That finding connects directly here: if the engine is rewriting your content and not naming your brand, measuring whether you were crawled tells you nothing about whether you are earning any discovery value.
Why Aggregate AI Visibility Scores Mislead
A first-hand measurement study across 1,790 data points demonstrated three ways aggregate AI visibility numbers produce false confidence:
Branded prompt inflation. One practitioner's dashboard showed 35% visibility and a #1 ranking. When branded prompts — queries that include the company name — were stripped out, organic visibility dropped to zero out of 96 prompts. The engines recognized the brand when asked about it directly but never surfaced it unprompted. Those are two entirely different problems, and an aggregate score hides the one that matters for growth.
Per-engine divergence. The same vendor ranked #1 on Perplexity, #3 on Google AI Overviews, and tied for #3 on ChatGPT search — in the same month, on the same topic. Each engine has a different content supply chain. Perplexity leans on what developers and communities currently recommend. AI Overviews inherit a decade of Google ranking signals. A single-number AI visibility score averages across engines that disagree and tells you nothing you can act on.
Technical optimization theater. After deploying the full on-site AI optimization stack — llms.txt, JSON-LD, entity schema — organic visibility went from 1/96 to 0/96 after 34 days. The engines cite what other sources say about you, not what you tell them about yourself.
Omniscient Digital made the distinction clearly: AI visibility is a diagnostic instrument, not a performance metric. Treating a 31% to 34% aggregate increase as progress is like reading a blood panel by averaging all the numbers into one score.
Four Metrics Worth Tracking Instead
If aggregate AI visibility scores are noise, here is what I am telling operators to measure:
1. Organic citation rate (unprompted queries only). Strip out every prompt that contains your brand name. What remains is the set of queries where a buyer asks a question before they have a shortlist. Your citation rate on those queries is the number that maps to discovery — the thing AI visibility was supposed to measure.
2. Per-engine citation share on buyer-intent prompts. Do not average across engines. Track ChatGPT, Perplexity, Google AI Overviews, and Claude separately on the same query set. The 11% cross-engine overlap data I covered earlier shows these are functionally different markets.
3. Named-brand citation rate. Of the queries where you are cited, how often does the AI actually name your brand in its answer text — versus burying a link in the footnotes? This is the ghost citation gap, and it separates brands that earn recognition from brands that supply raw material.
4. Referral traffic from AI engines (segmented). Measure actual sessions from ChatGPT, Perplexity, and Claude referrals independently. A study of 68.9 million AI crawler visits found that sites receiving AI crawler visits get 3.2x more human traffic — but the causation runs in the other direction. Content-rich, well-structured sites attract both crawlers and traffic. The crawler visit itself is a signal of site quality, not a cause of business value.
This Is an Earned Media Problem, Not a Crawler Metric
The brands that get named in AI answers — not just linked — are the ones that other authoritative sources already reference. The mechanism is the same one that drove traditional PR value, except now the audience includes AI retrieval systems that decide which brands to recommend in response to buyer questions.
This is what Machine Relations addresses as a discipline: earning AI citations is not a technical SEO problem solvable with server-log analysis or crawler optimization. It is an authority problem — making your brand legible, credible, and retrievable across the systems that are replacing search result pages as the primary discovery layer.
If your vendor is selling you AI crawler traffic as "visibility," ask them what percentage of those crawl hits produced a named citation in a live AI answer. The silence that follows will tell you whether you are measuring the wrong layer.
FAQ
How do I separate branded from organic AI visibility?
Run your prompt set with and without your brand name included. Branded prompts ("Is [Company] good at X?") test recall — whether the engine knows who you are. Organic prompts ("Best solution for X") test discovery — whether the engine thinks of you unprompted. Most AI visibility tools do not separate these, and the branded prompt artifact inflates scores significantly. Track both, report them separately.
Why do AI visibility scores differ so much between engines?
Each AI engine draws from different content pipelines. Perplexity indexes community recommendations and recent discussions. Google AI Overviews inherit legacy search ranking signals. ChatGPT search blends web retrieval with model training data. A brand that dominates one engine may be invisible on another. Measure per-engine, on the same prompt set, at regular intervals.
What is a ghost citation in AI search?
A ghost citation occurs when an AI engine links to your content as a source but never names your brand in the answer text. Research across 3,981 domains found 62% of AI citations are ghost citations — the brand supplies the evidence but receives no name recognition. The effective visibility drops from 53.1% to 10.6% when ghost citations are excluded.