AI Visibility Scores Shift Between Runs: What to Measure Instead
Research proves AI citation rankings shift between runs. Many score differences are statistical noise. Here's the measurement stack that works: inputs you control, not stochastic outputs.
Ask ChatGPT who leads your category. Ask again ten minutes later. You will get a different answer — and that is not a bug. It is how large language models generate citations, and it is the reason most AI visibility scores are misleading.
Ronald Sielinski's March 2026 study, "Quantifying Uncertainty in AI Visibility", is the first rigorous research to measure this across Perplexity, SearchGPT, and Gemini: citation distributions follow a power-law form, rankings shift between repeated samples, and many apparent differences between domains fall within what Sielinski calls the "noise floor of the measurement process." If your AI visibility strategy depends on a score that treats citations as fixed values, you are making decisions based on a snapshot of a distribution, not the distribution itself.
The practical implication: measure the causal inputs you control — earned media placements, entity consistency, structured data — not the stochastic outputs that fluctuate between API calls.
Key takeaways
- AI visibility scores are modeled estimates built from controlled testing, not measured from actual user behavior. No tool has access to real prompt data from ChatGPT, Perplexity, or Gemini.
- Citation rankings are unstable across repeated samples. The same query returns different cited domains at different times, even minutes apart.
- Bootstrap confidence intervals show that many "differences" between competing brands fall within statistical noise. What looks like a ranking shift may be meaningless.
- Forrester now calls the collapse of buyer research visibility the "visibility vacuum" and has made AI visibility the central theme of B2B Summit 2026.
- The measurement problem is a physics problem, not a tools problem. Non-deterministic systems cannot be measured with deterministic methods.
- Brands that focus on the causal input (earned media placements in trusted publications) rather than the non-deterministic output (citation counts) build durable visibility that persists across model updates and sampling variation.
Why are AI visibility scores non-deterministic?
Traditional search measurement works because Google exposes data about itself. Google Search Console reports impressions, clicks, average position from real user queries against a deterministic index. Position 4 means position 4. AI answer engines share none of this.
ChatGPT, Claude, Perplexity, and Gemini do not publish what users ask, how often they ask it, or which sources they consider for any given response. Forrester analyst John Buten described this directly: "Large language models don't share user data. They don't share what prompts people ask or how often those prompts are being asked."
Every AI visibility score on the market is constructed from synthetic prompts, not real user behavior. A tool sends its own queries to LLM APIs, records what comes back, and packages the result as a "score." The methodology determines the score more than the brand's actual visibility does.
A platform testing 50 prompts once a month produces a fundamentally different score than one running thousands of prompts daily across multiple models. Both call the output an "AI visibility score." Neither is measuring the same thing.
Research on LLM scoring inconsistency (2026) found that even within controlled evaluation tasks, large language models produce different numerical scores for the same input across runs, confirming that non-determinism is intrinsic to the architecture, not a fixable calibration error.
This asymmetry between what is measured and what is reported has concrete consequences for every founder evaluating AI visibility tools. When you see a competitor's score rise 15 points, you cannot determine whether their actual citation presence increased, whether the tool changed its prompt set, whether the sampling happened to catch a favorable moment, or whether all three factors combined.
What does the research show about citation variability?
Citation rankings shift across repeated samples of the same query, even minutes apart. Sielinski's study ran the same queries repeatedly across Perplexity Search, OpenAI SearchGPT, and Google Gemini using two sampling regimes: daily collections over nine days and high-frequency sampling at ten-minute intervals. Three findings should change how every founder evaluates AI visibility data.
First, citation distributions follow a power-law form. A small number of domains capture the majority of citations for any given query, while a long tail of domains appears sporadically. What is surprising is how unstable the power-law curve is across samples. The domain that ranked second in citations on Tuesday might rank fifth on Wednesday, not because anything changed about the domain, but because the model's response generation is stochastic.
Second, bootstrap confidence intervals reveal a wide noise floor. When Sielinski computed confidence intervals around citation share estimates, many "differences" between competing domains overlapped. A brand with an observed 12% citation share and a competitor at 9% may be statistically indistinguishable once error bars are applied. The numbers look different on a dashboard. They may mean nothing.
Third, rank stability is low across the full cited domain set. Even domains that appear frequently in citations experienced ranking instability across repeated samples. The paper concluded that "single-run visibility metrics provide a misleadingly precise picture of domain performance in generative search."
Research on AI agent reliability (Rabanser et al., 2026) evaluated 14 models across consistency, robustness, predictability, and safety dimensions. Recent capability gains have yielded only small improvements in reliability. Models are getting smarter but not more consistent, which means the measurement noise floor is not shrinking as models advance.
| AI platform | Citation behavior | Variability profile | Best for |
|---|---|---|---|
| Perplexity | Real-time web crawling, lower citation threshold | Higher volatility, shifts with new content | Smaller brands, recent content |
| ChatGPT | Training data + RAG weighted, conservative citations | More stable day-to-day, shifts on model updates | Established brands with publication history |
| Gemini | Own crawling/indexing, 23% more brand citations than ChatGPT | Moderate volatility | Brands with structured data |
| Claude | Highest accuracy, higher citation threshold | Fewer citations but more faithful descriptions | Brands with strong entity consistency |
Why does citation instability matter for revenue?
94% of B2B buyers now use AI during their purchasing process, making AI citation presence directly tied to revenue. This is not speculative.
Forrester's State of Business Buying 2026 report, surveying nearly 18,000 global business buyers, found that the typical buying decision includes 13 internal stakeholders and 9 external influencers. Buyers use AI for product research (54%), product comparisons (55%), evaluating RFP responses (48%), and building business cases (47%).
When 94% of prospective customers research through AI, citation presence directly affects whether you make the initial consideration set. And when that citation presence is measured with tools that cannot distinguish signal from noise, you are navigating a revenue-critical channel with an unreliable compass.
Forrester calls this broader collapse the "visibility vacuum": as buyer research shifts into answer engines, marketers lose visibility into what buyers asked, what content influenced them, and how decisions formed. You are not just losing traffic. You are losing the ability to understand the buying process.
Forrester's research on private AI adoption sharpens this further. More than half of business buyers use private AI tools provided by their employer, with Microsoft Copilot reaching 68% adoption among business buyers. The queries buyers run inside private AI instances are invisible to every commercial monitoring platform. Your AI visibility score cannot account for the majority of enterprise buying behavior it claims to measure.
What are the three structural measurement failures in AI visibility tools?
Every commercial AI visibility tool on the market makes at least one of these three failures. Most make all three.
1. Treating synthetic prompts as representative of real user behavior
No AI visibility tool has access to actual user prompts. They construct their own based on keyword research and assumptions about how buyers phrase questions. But LLM users do not search like Google users. They ask longer, more contextual questions. They reference prior conversation turns. They operate within company-specific AI instances behind firewalls.
Research on randomness in agentic evaluations found that most published AI benchmarks report scores from a single run per task, assuming reliability that does not exist. The same flaw carries over to commercial tools that run each prompt once and treat the output as ground truth.
2. Reporting point estimates without confidence intervals
A score of 72 out of 100 looks precise. If that score has a 95% confidence interval of [58, 86], the precision is illusory. Sielinski demonstrates that citation share estimates require repeated sampling and statistical treatment to produce interpretable numbers. Most commercial tools run each query once or a handful of times and report the result as deterministic.
3. Conflating visibility with accuracy
Being mentioned is not the same as being mentioned correctly. Research on LLM citation behavior (CiteAudit, 2026) found fabricated references in scholarly output at increasing rates, with hallucinated citations varying widely by model and domain. Cross-model audits of reference fabrication documented systematic citation hallucination patterns across major models. For SaaS companies with complex pricing tiers, an inaccurate AI mention can actively damage conversion by setting false expectations.
| Measurement approach | What it captures | What it misses |
|---|---|---|
| Single-run citation count | One snapshot of model output at one moment | Variability, confidence intervals, statistical significance |
| Multi-run with aggregation | Average citation presence across samples | Distribution shape, rank stability, prompt dependency |
| Bootstrap confidence intervals | Statistical bounds on true citation share | Prompt representativeness (still synthetic) |
| Cross-model comparison | Platform-specific citation behavior | Weighting by actual platform usage volume |
How do citation patterns differ across AI platforms?
Each AI platform selects and presents sources through architecturally different systems, producing fundamentally different citation behaviors for the same query. A blended score hides the information needed to form a platform-specific response strategy.
Surfaceable's 2026 AI Visibility Benchmark Report, tracking 60 brands across 20 industries, found that Gemini cites brands 23% more frequently than ChatGPT on commercial queries. Perplexity was the most likely platform to cite smaller brands, with a lower citation threshold than any other platform tested. ChatGPT showed the most conservative citation behavior, concentrating on well-established names.
Claude showed the highest accuracy scores among all platforms. Brands cited by Claude received more precise descriptions of their product and positioning, but Claude also applied a higher threshold for initial citation.
Sielinski's research on AI visibility variability adds another dimension: even within a single platform, citation behavior shifts over time in ways not attributable to any change the brand made. Perplexity's real-time crawling means a competitor publishing a new blog post can temporarily displace your citation presence, only for it to return hours later as the content pool rebalances. A tool measuring during that displacement reports a real drop that was never a real drop in underlying authority.
What actually predicts durable AI citation?
Earned media placements in trusted publications are the most stable predictor of AI citation persistence. Citation variability exists at the output layer because LLMs are stochastic systems, but the inputs that drive citation are far more stable.
Source authority and editorial placement
When multiple independent publications mention a brand in connection with a specific capability, LLMs develop a stronger prior for that association. This is the same mechanism that made earned media valuable for human audiences: third-party credibility in publications that buyers and models both trust.
AuthorityTech's research on earned media and AI search visibility documents how placements in trusted publications create citation persistence that single-run scores cannot capture. A Forbes feature or TechCrunch mention does not fluctuate between sampling runs because it exists as a permanent node in training and retrieval data. As Jaxon Parrott has written, founders who build for the AI citation market create structural advantage that compounds regardless of measurement noise.
Entity consistency across sources
Surfaceable's benchmark found that consistent brand descriptions across review platforms (G2, Capterra, Trustpilot, Crunchbase) correlated strongly with visibility scores above 75 out of 100. Research on citation attribution in LLMs (CiteGuard, 2025) found that retrieval quality and source grounding strongly affect generated output quality. Brands with fragmented entity descriptions give models conflicting signals, resulting in lower citation confidence. A study on aligning LLM citation behavior with human preferences found models are 27% more likely to cite content explicitly flagged as needing citations, while under-citing content with personal names and specific numbers.
Structured data and topical depth
Surfaceable's analysis found that mid-market B2B SaaS companies regularly outperformed Fortune 500 companies in AI citation performance. Smaller companies had invested specifically in structured, answer-led content architecture (topic clusters, FAQ schema, Organization schema), while enterprise brands relied on domain authority that does not automatically translate to AI citation.
The signals with strongest correlation to scores above 75: structured data on key pages (Organization, FAQPage, Article schema), FAQ-format content addressing target queries, Wikipedia or Wikidata entity presence, active blog publishing, verified Google Business Profile, consistent brand descriptions across review platforms, and full AI crawler access with no robots.txt restrictions blocking GPTBot, ClaudeBot, or PerplexityBot.
| Signal type | Stability across sampling | Measurability | Control level |
|---|---|---|---|
| Single-run citation score | Low (varies between runs) | Easy (any tool reports it) | None (cannot influence stochastic output) |
| Earned media placement count | High (permanent once published) | Medium (requires tracking placements) | High (controlled through editorial relationships) |
| Entity consistency score | High (stable across platforms) | Medium (requires cross-platform audit) | High (brand controls descriptions) |
| Structured data presence | High (deterministic, crawlable) | Easy (automated schema checks) | High (brand implements directly) |
| Content topical depth | High (content is persistent) | Medium (topic cluster analysis) | High (editorial investment) |
How should you build a measurement stack that actually works?
The practical response to measurement unreliability is not to stop measuring. It is to change what you measure and how you interpret it.
Track inputs, not just outputs. Count earned media placements per quarter. Track publications mentioning your brand in connection with target queries. These numbers are deterministic. They do not fluctuate between API calls. When the input grows, the output follows, even if the output is noisy on any individual measurement.
Demand statistical treatment from your tools. Any AI visibility platform that reports a single number without a confidence interval or sample size is giving you a weather report based on one thermometer reading. Ask how many times each prompt was run. Ask whether results include error bars. If the answer is "we run it once," the number is anecdotal.
Measure trends, not snapshots. A single AI visibility score is noise. A directional trend across 60 or 90 days of repeated measurement starts to become signal. If your share of citation is consistently increasing across multiple prompt sets and platforms over three months, that is meaningful. If it jumped 8 points between last Tuesday and this Tuesday, that is probably the noise floor.
Separate platform behavior. Perplexity's aggressive real-time web crawling shows higher citation volatility than ChatGPT, which weights training data authority more heavily. Measure each platform independently and weight by relevance to your buyers.
Validate accuracy, not just presence. Check what AI engines actually say about your brand, not just whether they mention you. Build a quarterly audit that reviews the top 10 prompts your buyers are likely to use and records what each engine says, word for word. An inaccurate mention is worse than no mention.
Invest in the causal layer. The brands with the most durable AI citation presence are the ones with the deepest editorial presence in publications AI engines trust. Every placement in Forbes, TechCrunch, or Harvard Business Review is a permanent node in the data these models use. That node does not fluctuate between runs. This is the core insight behind tracking AI traffic attribution: measure what you can control.
Frequently asked questions about AI visibility score accuracy
Are AI visibility scores completely useless?
No. They provide directional signal when interpreted correctly. A score treated as an approximate trend indicator over months of repeated measurement has value. A score treated as a precise ranking on a single day does not. The problem is false precision, not the scores themselves.
How many samples does an AI visibility measurement need?
Single-run estimates are inadequate for most use cases. The minimum depends on the granularity of comparison: distinguishing a 60% citation share from a 20% share requires fewer samples than distinguishing 35% from 30%. Sielinski's research provides guidance on sample sizes required for interpretable confidence intervals.
Which AI platform is most stable in its citations?
ChatGPT shows the most stable citation patterns because it relies more heavily on training data than real-time crawling. Perplexity has the highest volatility because its source pool changes continuously. Gemini falls between the two. No platform is fully stable because all use non-deterministic generation.
What should I tell my board about AI visibility?
Report on inputs you control (earned media placements, entity consistency, structured data coverage) and present AI citation data as directional trends with explicit caveats about measurement precision. Do not present a single AI visibility score as equivalent to a search ranking. The board needs to understand that AI visibility is a distribution, not a number.
Does earned media actually improve AI citations?
Yes. Placements in publications that AI engines index and trust create persistent citation signals that survive model updates and sampling variation. This is the core mechanism of Machine Relations: the same earned media strategy that built brand credibility with human readers now builds citation presence with AI readers. PR's original mechanism, earning placement in respected publications through editorial relationships, is the most stable input to AI citation.
The AI visibility market will mature. Tools will improve. Statistical rigor will become table stakes. But the brands best positioned when measurement catches up are the ones building the causal infrastructure now: editorial relationships with publications AI engines trust, entity consistency across every platform, and content depth that makes their expertise unambiguous to any system that encounters it.
The measurement problem is real. The brands solving it are not waiting for better scores. They are building the editorial presence that makes the score inevitable.