Everyone Agrees AI Rankings Are Noise. Nobody Is Asking the Right Next Question.

The AI visibility measurement industry just published the evidence that its own product does not work. Two peer-reviewed papers, SparkToro's study showing AI recommendations change more than 99% of the time, and Search Engine Journal's July 11 analysis all arrive at the same conclusion: the number on your AI visibility dashboard is a single sample from a distribution, not a measurement of where you stand. The industry's response has been to measure more often. That is the wrong response. The right one is to question whether "ranking" was ever the correct frame.

The Research Is Now Replicated. This Is Not One Study.

In March, Ronald Sielinski published "Quantifying Uncertainty in AI Visibility", showing that citation shares across Perplexity, SearchGPT, and Gemini follow a power-law form and that many apparent differences between competing domains fall within the noise floor. In April, the same research group released "Don't Measure Once: Measuring Visibility in AI Search", testing 30 platform-topic combinations across three AI engines.

The findings are concrete. Rankings required between 33 and 94 citation-bearing answers before they stabilized. Three of the 30 tests never stabilized at all, even after 125 questions. All three were on SearchGPT. The typical margin of error for a top-10 cited domain was roughly five positions, and one in five margins were wider than ten.

Take one of the examples: Tom's Guide appeared in about 9.5% of SearchGPT running-gear citations while Runner's World appeared in about 6.0%. On a dashboard, Tom's Guide "outranks" Runner's World. But the 3.5-point gap fell within the margin of error. With a single sample, you cannot say one outperforms the other. The number was real. The conclusion drawn from it was not.

The Industry Reaction Tells You Everything

Watch how the market responded. Tool vendors started talking about "more frequent measurement" and "confidence intervals." Rand Fishkin's advice: make sure your provider "shows their math." The researchers themselves proposed stopping rules to determine when you have enough data.

All reasonable. All missing the point.

The question everyone is trying to answer is: how do we measure AI visibility ranking more accurately? But that assumes "ranking" is the right measurement to refine. It is not.

Google had rankings because Google had a list. Ten blue links, ordered by algorithm, same for every user in the same geography. Position was a real property of a real system. You could measure it because it existed.

AI engines do not have a list. They have a source selection process that fires differently on every query, shaped by prompt phrasing, session context, model temperature, and the probabilistic nature of token generation itself. "Position" is not a noisy measurement of something real. It is a fictional construct applied to a system that never had positions to begin with.

Trying to stabilize AI visibility into a ranking is like measuring the exact location of a particle that exists as a probability wave. The uncertainty is not a flaw in the instrument. It is a property of the system.

What Is Actually Stable (And Worth Measuring)

Three things hold still when you stop looking for a rank that does not exist.

Source selection frequency. Across enough queries in your category, are you being cited at all? Not where you are on a leaderboard. Whether you are on the field. This is binary at the query level and statistical at the category level, and it converges much faster than positional ranking does.

Citation context. What are you being cited for? A machine selecting you as background context for a comparison table is different from a machine quoting your data as the primary answer. The value of a citation is not the citation itself. It is what the citation proves the machine believes about you.

The inputs you control. Published evidence in authoritative outlets. Structured data that resolves your entity without ambiguity. Claims grounded in specific numbers from primary sources. These are the causal inputs to source selection. They do not shift between API calls because they are not stochastic outputs. They are the architecture the stochastic process draws from.

I wrote about this shift in April, before the second paper or the SEJ coverage. The thesis was the same: the measurement problem is a physics problem, not a tools problem. Non-deterministic systems cannot be measured with deterministic methods. Three months later, the research has replicated and the industry has reached consensus on the diagnosis while still prescribing the wrong treatment.

The Frame Problem Is the Machine Relations Problem

This is exactly the territory Machine Relations was built for. Traditional PR measured reach and impressions. SEO measured rankings. Both assumed a stable, observable position in a fixed output. AI source selection does not work that way, and the discipline that replaces those older models has to be built on a different measurement foundation: not where you appear, but whether the machine treats you as a credible source worth selecting.

Every vendor selling you an "AI rank tracker" is applying a Google-era measurement concept to a system that is architecturally incompatible with it. The tool is not broken. The category is wrong.

FAQ

How many times do I need to query an AI engine before the results mean something?

The research tested 30 platform-topic combinations and found you need between 33 and 94 citation-bearing answers before rankings stabilize, and three tests never stabilized even after 125 queries. The exact number depends on the engine and topic. There is no universal cutoff.

Should I stop tracking AI visibility entirely?

No. Stop tracking positional rank. Track source selection frequency across query categories, the context in which you are cited, and the inputs you control: authoritative placements, structured data, and entity consistency. Those measurements converge and they reflect the things you can actually change.

Why do different AI engines need different sample sizes?

Gemini concentrates citations on a few domains within each answer, so many of its citations carry overlapping information. SearchGPT spreads citations across more sources, meaning each answer carries more independent signal. The same number of queries on two engines does not buy the same confidence.

Additional source context

• Measuring Visibility · Visibility Kit Docs # Measuring Visibility Recall and grounded channels, across the engines people use, counted by frequency. (Measuring Visibility · Visibility Kit Docs (visibilitykit.ai)).
• AI Visibility Rankings Aren't Stable – New Research Shows It's Mostly Statistical Noise AI visibility tracking data isn’t entirely reliable. (AI Visibility Rankings Aren't Stable – New Research Shows It's Mostly Statistical Noise (searchenginejournal.com), 2026).
• Despite this stochastic behavior, current approaches to measuring domain visibility in generative search typically rely on single-run point estimates of citation share and prevalence, implicitly treating them as fixed values. (Quantifying Uncertainty in AI Visibility: A Statistical Framework for Generative Search Measurement (doi.org), 2026).