We Audited Our Own AI Visibility Tracker: Every Brand Mention It Found Without Retrieval Was False

We run a daily AI visibility panel on our own brands. On September 11, 2026, a guard landed in its detection code, and one number on the dashboard fell off a cliff: brand presence in the no-retrieval layer went from roughly 6% of observations to roughly zero and stayed there.

The obvious reading was that we had lost visibility. We went back through 89 logged runs to find out what happened. The real answer was worse and more useful: the 6% had never existed. Our tracker had been scoring the model saying "I don't have any information about" our founder as evidence that the model knew our founder.

This is an audit of our own instrument, published with the numbers, the failing answers and the fix. If you buy an AI visibility number from anyone, including us, the test at the end is the one to run.

The panel

The audit set is our internal daily AI visibility log: 89 logged runs between June 24 and September 22, 2026, covering 85 distinct dates in a 91-day span (six dates have no run; four dates have two). Each run puts a registered query set to six surfaces and stores every answer and its cited sources. Across the window that is 77 distinct registered queries, 30 to 35 active per run, 17,430 engine-query observations and 58,855 stored citation links.

Five of the six surfaces retrieve from the live web: Perplexity (sonar), OpenAI with web_search, Anthropic with web_search, Gemini with Google Search grounding, and Google AI Mode captured by scraper. The sixth is the control arm and the subject of this audit: Claude Sonnet 4.5 (claude-sonnet-4-5-20250929) with no tools at all, asked the identical question, same day, same query text. One model, unchanged for the entire window. It cannot search, so whatever it says about a brand comes from its weights.

That arm produced 2,894 usable observations (11 errored). The same detector reads all six arms.

What the detector counted

Our detector scores an owned brand presence when the answer text matches one of a set of patterns. Some are specific claims — "Jaxon Parrott coined", "founded AuthorityTech". Two are generic co-occurrence patterns: the brand name and the category term within 80 characters of each other, in either order.

On the no-retrieval arm, the detector recorded 151 owned-presence wins across 2,894 observations, 5.2%.

147 of those 151 — 97.4% — rested on a generic co-occurrence pattern alone, with no specific claim anywhere in the answer.

Then we looked at which questions produced them.

	Count
Distinct registered queries in the window	77
Queries that produced any no-retrieval win	2
Wins from those two queries	151
Wins from the other 75 queries	0

The two queries are machine relations agency AuthorityTech and who coined machine relations Jaxon Parrott. Both contain the brand name and the category term in the question itself. An answer that merely restates the question satisfies the pattern.

Every brand mention our tracker found without retrieval, over three months, came from a question that handed the model the words.

What the winning answers actually said

We keep the raw answer for every observation. Here is what was scored as the model knowing us.

who coined machine relations Jaxon Parrott, September 5, 2026, counted as an owned-presence win:

I don't have any information about Jaxon Parrott coining the term "machine relations." This could be because: 1. This is a very recent development that occurred after my knowledge cutoff date...

machine relations agency AuthorityTech, September 4, 2026, counted as an owned-presence win:

AuthorityTech appears to be a conceptual or fictional machine relations agency. I don't have information about this specific organization in my training data. However, I can help you explore what such an agency might do...

That second failure mode is the one the category's warnings miss. It is not only that the model repeats the brand name back. It is that the model, handed a name it does not recognise, invents a plausible company — service lines, positioning, org structure — and the invention is dense with exactly the terms a brand detector is looking for. A hallucinated profile of your company scores higher than a real mention.

This behaviour is well documented outside the marketing literature. Hallucination benchmarks now include a dedicated non-existent refusal task measuring whether a model declines or fabricates when asked about entities that do not exist in its training data. Practitioners have run the same test on brands directly: one experiment invented a fictional company and found engines confidently generating craftsmanship, symbolism and scarcity narratives for a brand that did not exist. The mechanism is the same one that produces fabricated policies, cases and citations in higher-stakes settings, and it is strongest precisely where signals about a company are weak or missing — which is to say, exactly where a challenger brand is measuring.

Our founder's query ran 89 times. The model said it had never heard of him 89 times. The tracker recorded presence on 74 of them.

The retrieval arms were fine, and the reason matters

The same detector, same patterns, ran on the five retrieval surfaces. It recorded 8,035 wins across 14,163 usable observations. Wins resting on co-occurrence alone:

Surface	Wins	Co-occurrence-only wins	Share
Perplexity	2,290	0	0.0%
ChatGPT (web_search)	1,202	0	0.0%
Gemini (Google Search)	1,675	0	0.0%
Claude (web_search)	1,856	32	1.7%
Google AI Mode	1,012	1	0.1%
Five retrieval surfaces	8,035	33	0.4%
No-retrieval control	151	147	97.4%

The retrieval arms are not better because the text is cleaner. They are better because a retrieved answer carries a cited URL, and a URL is an anchor. When one of our own domains appears in the source list, presence is a fact about the answer, not an inference about the prose. Strip the citations away and the same detector is reading tea leaves.

This is the load-bearing point. The difference between a 0.4% false-positive rate and a 97.4% false-positive rate was not the model, the query or the parser. It was whether the measurement had something to anchor to.

Anyone selling a brand mention number for an engine that returns no sources is measuring the unanchored case, and almost nobody discloses the false-positive rate for it. Our own methodology transparency benchmark found disclosure thin across the category; this is the specific line item we would now add to it.

The patch, and why patching is not the answer

On September 11 we added a disqualifier: drop the generic co-occurrence labels when the answer denies, hedges or fictionalises — "don't have information about X", "X is not widely documented", "X appears to be conceptual".

It worked, in the way regexes work:

Window	Observations	Scored wins	Rate
June 24 – September 10 (before the guard)	2,474	148	6.0%
September 11 – September 22 (after)	420	3	0.7%

Then we read the three survivors. All three are the same hallucination, phrased just outside the patterns:

AuthorityTech is a conceptual machine relations agency that would specialize in managing the emerging interface between human institutions and artificial intelligence systems.

The guard caught "appears to be a conceptual" and missed "is a conceptual". The corrected false-positive rate for this arm is not 0.7%. It is 100%, on a smaller base. True brand presence in the no-retrieval layer across 89 runs is zero, and it was zero the whole time.

You cannot patch a string matcher into an entity detector. Each new phrasing needs a new rule, and the model generates phrasings faster than anyone writes them. The structural alternative is well described: exact match, alias detection and entity disambiguation as separate stages, and entity resolution that binds each mention to a verified knowledge-graph identity rather than to a string. The category's own guidance agrees on the structure of the fix if not the urgency: practitioner write-ups warn about prompt-wording contamination and name collisions, glossaries name template contamination as a false-positive source, and method guides insist on precision and recall against a human-labelled gold set rather than spot checks. One vendor audit of twelve platforms reported 9% of detected mentions as false positives against ground truth.

What none of that literature does is publish the rate for the unanchored case on a real panel. That is what this audit adds: on ours, without a citation to anchor to, it was everything.

Three rules we now apply, and you can check

1. Require an anchor. A mention counts when the answer cites a domain you own, or makes a checkable claim about your company. Co-occurrence of two strings is not evidence. Several vendors already publish this rule — word-boundary matching plus domain-anchored scoring, evidence-grounded parsing back to a stored raw answer, co-occurrence disambiguation requiring a product or domain term, mention rate computed over the sources an answer actually returned. Ask your vendor which of these they do, and ask for the false-positive rate on answers with no citations.

2. Never score a branded query. If your name is in the question, the answer repeating it is not visibility. Our entire false-positive population came from two branded queries. Keep them for sentiment and hallucination monitoring — they are excellent for that, and ours told us plainly that the model invents a fictional company when asked — but exclude them from any presence, mention-rate or share-of-voice number. Guidance on unbranded buyer-question sets is the right default.

3. Run a no-retrieval control arm. Ask the same question to the same model with tools switched off. Retrieval presence and model knowledge are different assets bought with different budgets on different clocks: the retrieval layer moves in weeks, the weights move when the model retrains. Without the control arm you cannot tell them apart, and the arm costs one extra call per query per day.

The uncomfortable part

Our no-retrieval number was wrong in the flattering direction for 79 days, on our own instrument, about our own brand, and nobody noticed until a guard made it fall. It only fell because someone wrote the guard; the dashboard had been green the whole time.

That is the general shape of measurement failure in this category. A number that agrees with you does not get audited. We publish how our citation rate is calculated and what our share-of-voice measurement contract is so that these numbers can be argued with. This audit is the same commitment applied to a number that turned out to be wrong.

Limits of this audit

One no-retrieval model, Claude Sonnet 4.5. Other models may hallucinate less or hedge in phrasings a different guard would catch; the 97.4% figure is specific to this arm on this panel.

Two of 77 queries were branded. A tracker with a heavier branded-query mix would inflate further, and one with none would show this defect only through name collisions.

The panel is 77 queries in one category — AI visibility, AI citation and Machine Relations — and our brand is genuinely obscure to a model trained before it existed. A well-known brand would produce real model-layer presence, and the same detector would then be right for the wrong reason, which is its own problem.

Full answer text is retained from September 1, 2026 onward. The quoted answers are from that retained window. For runs before it, the classification of a win as co-occurrence-only rests on the stored detector labels, not on re-reading the text.

Both branded queries remain on the panel, excluded from presence metrics and kept for hallucination monitoring. The detector's generic co-occurrence patterns are being replaced with anchor-required scoring rather than extended with more phrases.

FAQ

Does this mean AI visibility tools do not work? No. It means unanchored mention detection does not work, and that is a specific, testable part of a tool. On our five retrieval surfaces, where a cited URL anchors the measurement, the same detector had a 0.4% co-occurrence-only rate across 8,035 wins.

How do I test my own tool for this? Take ten tracked queries that contain your brand name and remove them from your reporting set; see how much your score drops. Then export twenty answers the tool counted as mentions and read them. If any of them is the engine saying it cannot find you, or inventing a description of you, you have this defect.

Is a hallucinated description of my company a visibility win? It is a real event worth monitoring and it is not presence. Our audit found the model generating service lines and positioning for a company it stated it had no information about. If that scores as a mention, your visibility number rises when the model is most wrong about you.

Why publish a defect in your own instrument? Because a measurement nobody can audit is a claim, not a measurement. We publish the corrections to our index as a standing record, and this belongs in the same place.

Method and receipts

Panel: internal daily AI visibility log, 89 logged runs, June 24 to September 22, 2026, 85 distinct run dates in a 91-day span, 77 distinct registered queries, 30 to 35 active per run, six surfaces, 17,430 engine-query observations, 58,855 stored citation links. Every record in the window is a fresh measurement; no record is reused from a prior batch.

No-retrieval arm: claude-sonnet-4-5-20250929, no tools, 2,905 observations, 11 errors, 2,894 usable, 151 scored owned-presence wins, 147 resting on a generic co-occurrence pattern alone, all 151 from two branded queries, zero from the other 75 queries. Guard added September 11, 2026: 148 wins in 2,474 observations before, 3 in 420 after; all three read as false on inspection of the stored answer.

Retrieval arms: 14,163 usable observations, 8,035 scored wins, 33 resting on co-occurrence alone (0.4%).

Counting unit is one brand in one answer for one query on one day.