How to Tell If Your AI Visibility Tool Is Counting False Brand Mentions

A brand mention counted from prose alone is not evidence that an AI engine knows you. It is evidence that two strings appeared near each other — which also happens when an engine restates the question it was asked, disclaims knowledge of the entity, or improvises a description of it.

There are three tests that separate a real AI visibility number from a decorative one. Require an anchor. Never score a branded query. Run a no-retrieval control arm. Each takes under an hour, none needs vendor cooperation, and the rest of this page is the first-party evidence for why they are the right three.

That evidence comes from the no-retrieval control arm we run on our own panel — the instrument that isolates the effect, and one we have not seen a tracker publish.

The three tests

1. Require an anchor. A mention counts when the answer cites a domain you own, or makes a checkable claim about your company — "X was founded by", "X coined", "X's product does". Proximity of your brand name to your category term is not a claim and should never score. This is the rule our own reporting runs on. Pieces of it appear in vendor documentation — word-boundary matching plus domain-anchored scoring, evidence-grounded parsing back to a stored raw answer, co-occurrence disambiguation requiring a product or domain term, mention rate computed over the sources an answer actually returned — so ask which of them your vendor does, then ask the question we have not found published anywhere: what is your false-positive rate on answers that return no sources?

2. Never score a branded query. If your name is in the question, an answer repeating it is not visibility. Branded queries are excellent for sentiment and hallucination monitoring and useless for presence. Exclude them from every presence, mention-rate and share-of-voice figure. Unbranded buyer-question sets are the right default. Our panel carries two branded queries out of 77, excluded from every presence figure and retained for hallucination monitoring. On the control arm those two produced every unanchored win across three months — the clearest query-design effect we have measured, and the reason the exclusion is a rule rather than a preference.

3. Run a no-retrieval control arm. Put the identical question to a frozen model with tools switched off. Retrieval presence and model knowledge are different assets, bought with different budgets, on different clocks: the retrieval layer moves in weeks, the weights move only when the model retrains. No tracker we have reviewed sells a visibility number for a tool-less model, and that is exactly why it is useful as an instrument: it is the clean case where the detector has no citations to lean on, so it exposes what your scoring logic does when the anchor is removed. Every retrieval tool hits that same unanchored case whenever an answer returns thin or no sources. The control arm costs one extra API call per query per day and tells you, in advance, how much of your number is made of prose.

Test three is how we isolated the effect below.

The panel

Our internal daily AI visibility log: 89 logged runs between June 24 and September 22, 2026, covering 85 distinct dates in a 91-day span (six dates have no run; four dates have two). Each run puts a registered query set to six surfaces and stores every answer and its cited sources. Across the window that is 77 distinct registered queries, 30 to 35 active per run, 17,430 engine-query observations and 58,855 stored citation links.

Five of the six surfaces retrieve from the live web: Perplexity (sonar), OpenAI with web_search, Anthropic with web_search, Gemini with Google Search grounding, and Google AI Mode captured by scraper. The sixth is the control arm: Claude Sonnet 4.5 (claude-sonnet-4-5-20250929) with no tools at all, asked the identical question, same day, same query text, one frozen model for the entire window. It cannot search, so whatever it says about a brand comes from its weights. It produced 2,894 usable observations (11 errored).

One detector reads all six arms. That is the whole experiment: hold the scoring method constant and vary only whether the answer carries citations.

What generic co-occurrence matching counts

The scoring method under test records an owned brand presence when the answer text matches one of a set of patterns. Some are specific claims — "[founder] coined", "founded [company]". Two are generic co-occurrence patterns: the brand name and the category term within 80 characters of each other, in either order. Those two generic patterns are the variable this page is about; holding them constant across all six arms is what makes the comparison below readable.

On the no-retrieval arm, the detector recorded 151 owned-presence wins across 2,894 observations, 5.2%.

147 of those 151 — 97.4% — rested on a generic co-occurrence pattern alone, with no specific claim anywhere in the answer. The remaining four matched a specific-claim pattern; we return to them in the limits, because they are the only ones that could have been real.

Then we looked at which questions produced them.

	Count
Distinct registered queries in the window	77
Queries that produced any no-retrieval win	2
Wins from those two queries	151
Wins from the other 75 queries	0

Both queries contain a tracked brand name and the category term in the question itself. An answer that merely restates the question satisfies the pattern.

That concentration is the finding, not a caveat on it. Two queries out of 77 were enough to generate an entire unanchored metric — which is why branded queries belong outside presence reporting rather than merely flagged inside it. A scoring defect does not need to be widespread to be total.

What the scored answers looked like

We keep the raw answer for every observation. Read in the retained window, the scored no-retrieval answers fell into two shapes, both of which satisfy a proximity pattern:

• Restatement with a disclaimer. The model repeats the brand and category from the question and says it cannot confirm anything about them. Every term the detector wants is present, because the question supplied it.
• Improvised description. Handed a name it cannot place, the model generates a plausible profile — service lines, positioning, org structure — dense with exactly the terms a brand detector looks for. An improvised profile scores higher than a real mention.

That second shape is the one the category's warnings miss. It is well documented outside the marketing literature. Hallucination benchmarks now include a dedicated non-existent refusal task measuring whether a model declines or fabricates when asked about entities absent from its training data. Practitioners have run the same test on brands directly: one experiment invented a fictional company and found engines confidently generating craftsmanship, symbolism and scarcity narratives for a brand that did not exist. The mechanism is the same one that produces fabricated policies, cases and citations in higher-stakes settings, and it is strongest wherever signals about a company are weak in the model's weights — which, for a frozen model, includes every company and category that grew after its training cutoff.

Full answer text is retained from September 1, 2026; for earlier runs the classification rests on stored detector labels rather than a re-read.

The anchor is the variable, and that is the transferable law

The same detector, same patterns, ran on the five retrieval surfaces. It recorded 8,035 wins across 14,163 usable observations. Wins resting on co-occurrence alone:

Surface	Wins	Co-occurrence-only wins	Share
Perplexity	2,290	0	0.0%
ChatGPT (web_search)	1,202	0	0.0%
Gemini (Google Search)	1,675	0	0.0%
Claude (web_search)	1,856	32	1.7%
Google AI Mode	1,012	1	0.1%
Five retrieval surfaces	8,035	33	0.4%
No-retrieval control	151	147	97.4%

The retrieval arms are not better because the text is cleaner. They are better because a retrieved answer carries a cited URL, and a URL is an anchor. When one of our own domains appears in the source list, presence is a fact about the answer, not an inference about the prose. Strip the citations away and the same detector is reading tea leaves.

Note what is held constant in that table. Claude appears twice — web_search on and tools off. Same model family, same day, same query text, same parser. The co-occurrence-only share moves from 1.7% to 97.4%. The variable is not the model, the query or the parser. It is whether the measurement had something to anchor to.

That generalises past the control arm. Any answer that returns thin or no sources puts a retrieval tool in the unanchored case, and almost nobody discloses what their scoring does there. Our own methodology transparency benchmark found disclosure thin across the category; the unanchored false-positive rate is the specific line item we would now add to it.

This is a Machine Relations measurement principle before it is a tooling one. Whether a machine knows your company is a claim about an entity, and entity claims need identity, not string proximity. A number that cannot distinguish "an engine cited your domain" from "an engine wrote your name twice while telling a user you do not exist" is not measuring the relationship at all.

Why a phrase guard does not fix it

On September 11 we added a disqualifier to the control arm: drop generic co-occurrence labels when the answer disclaims or improvises — phrasings such as "don't have information about", "is not widely documented", "appears to be conceptual".

Window	Observations	Scored wins	Rate
June 24 – September 10 (before the guard)	2,474	148	6.0%
September 11 – September 22 (after)	420	3	0.7%

We read all three survivors. Each was the same improvised-description shape, phrased one word outside the guard — "is a conceptual" where the guard caught "appears to be a conceptual". Three wins, three read, three unanchored. The guard lowered the count; it did not change what the detector was measuring. Across the full window, no no-retrieval win we have read survives inspection, and the concentration of all 151 in two branded queries is why we do not expect the unread ones to; the limits below say exactly how far that inference reaches.

You cannot patch a string matcher into an entity detector. Each new phrasing needs a new rule, and the model generates phrasings faster than anyone writes them. The structural alternative is well described: exact match, alias detection and entity disambiguation as separate stages, and entity resolution that binds each mention to a verified knowledge-graph identity rather than to a string. The category's own guidance agrees on the structure of the fix if not the urgency: practitioner write-ups warn about prompt-wording contamination and name collisions, glossaries name template contamination as a false-positive source, and method guides insist on precision and recall against a human-labelled gold set rather than spot checks. One vendor audit of twelve platforms reported 9% of detected mentions as false positives against ground truth.

What none of that literature does is publish the unanchored case on a real panel with a control arm. That is what this adds: without a citation to anchor to, not one scored win we read survived inspection.

The standard this holds us to

A number that agrees with you does not get audited, and that is the general shape of measurement failure in this category. The control arm exists so our own numbers are audited whether they flatter us or not. We publish how our citation rate is calculated and what our share-of-voice measurement contract is so that every number can be argued with. A measurement nobody can audit is a claim, not a measurement.

Limits

One no-retrieval model, Claude Sonnet 4.5. Other models may hallucinate less or hedge in phrasings a different guard would catch; the 97.4% co-occurrence-only share is specific to this arm on this panel.

Two of 77 queries were branded. A tracker with a heavier branded-query mix would inflate further, and one with none would show this defect only through name collisions rather than prompt echo.

The panel is 77 queries in one category — AI visibility, AI citation and Machine Relations — and a frozen model's weights lag any category that is still being defined on the live web. A brand heavily represented in training data would produce real model-layer presence, and the same detector would then be right for the wrong reason, which is its own problem.

Full answer text is retained from September 1, 2026 onward. The answer shapes described above are from that retained window. For runs before it, the classification of a win as co-occurrence-only rests on the stored detector labels, not on re-reading the text.

Co-occurrence-only share and false-positive rate are two different numbers, and this page reports both. The 97.4% and 0.4% figures are co-occurrence-only shares, computed from stored detector labels across every scored win. A false-positive rate requires reading the answer behind the label, and we report one only where we did that: 100% on the three post-guard survivors. We have not re-read the 33 co-occurrence-only wins on the retrieval arms, so 0.4% bounds their false-positive rate from above rather than measuring it. A page arguing that unread labels are not evidence has to hold itself to the same line.

Four of the 151 no-retrieval wins matched a specific-claim pattern rather than a co-occurrence pattern, and we have not inspected them individually. They are the wins most likely to reflect something the model actually held, and the case that the no-retrieval layer contained no true presence rests on the concentration of all 151 in two branded queries, not on having read all four.

Both branded queries remain on the panel, excluded from presence metrics and kept for hallucination monitoring. The generic co-occurrence patterns are being retired in favour of anchor-required scoring rather than extended with more phrases: test one is the rule we hold ourselves to, not only the one we recommend.

FAQ

Does this mean AI visibility tools do not work? No. It means unanchored mention detection does not work, and that is a specific, testable part of a tool rather than a verdict on the category. On our five retrieval surfaces, where a cited URL anchors the measurement, the same detector had a 0.4% co-occurrence-only rate across 8,035 wins.

Isn't a model with no retrieval the wrong thing to measure AI visibility on? It is the wrong thing to report and the right thing to test with. Nobody should sell a visibility number for a tool-less model. But it is the only clean way to see what your scoring logic does when there are no citations in the answer — and retrieval engines return thin or empty source lists often enough that every tool meets that case in production.

How do I test my own tool for this? Take ten tracked queries that contain your brand name and remove them from your reporting set; see how much your score drops. Then export twenty answers the tool counted as mentions and read them. If any of them is the engine disclaiming knowledge or improvising a description, you have this defect.

Is a hallucinated description of my company a visibility win? It is a real event worth monitoring and it is not presence. On the control arm, improvised profiles — service lines and positioning generated without a source — were the most detector-friendly text the model produced. If that scores as a mention, your visibility number rises exactly when the model is guessing.

Method and receipts

Panel: internal daily AI visibility log, 89 logged runs, June 24 to September 22, 2026, 85 distinct run dates in a 91-day span, 77 distinct registered queries, 30 to 35 active per run, six surfaces, 17,430 engine-query observations, 58,855 stored citation links. Every record in the window is a fresh measurement; no record is reused from a prior batch.

No-retrieval arm: claude-sonnet-4-5-20250929, no tools, 2,905 observations, 11 errors, 2,894 usable, 151 scored owned-presence wins, 147 resting on a generic co-occurrence pattern alone and 4 on a specific-claim pattern, all 151 from two branded queries, zero from the other 75 queries. Guard added September 11, 2026: 148 wins in 2,474 observations before, 3 in 420 after; all three read as false on inspection of the stored answer.

Retrieval arms: 14,163 usable observations, 8,035 scored wins, 33 resting on co-occurrence alone (0.4%), none individually re-read.

Shares of co-occurrence-only wins are computed from stored detector labels. Inspected false-positive counts are stated separately and only where the stored answer text was read.

Counting unit is one brand in one answer for one query on one day.