How to Tell If Your AI Visibility Tool Is Counting False Brand Mentions

A brand mention counted from prose alone is not evidence that an AI engine knows you. It is evidence that two strings appeared near each other — which also happens when an engine restates the question it was asked, disclaims knowledge of the entity, or improvises a description of it.

There are three tests that separate a real AI visibility number from a decorative one. Require an anchor. Never score a branded query. Run a no-retrieval control arm. Each takes under an hour, and none needs vendor cooperation.

The three tests

1. Require an anchor. A mention counts when the answer cites a domain you own, or makes a checkable claim about your company — "X was founded by", "X coined", "X's product does". Proximity of your brand name to your category term is not a claim and should never score. Pieces of it appear in vendor documentation — word-boundary matching plus domain-anchored scoring, evidence-grounded parsing back to a stored raw answer, co-occurrence disambiguation requiring a product or domain term, mention rate computed over the sources an answer actually returned — so ask which of them your vendor does, then ask the question we have not found published anywhere: what is your false-positive rate on answers that return no sources?

2. Never score a branded query. If your name is in the question, an answer repeating it is not visibility. Branded queries are excellent for sentiment and hallucination monitoring and useless for presence. Exclude them from every presence, mention-rate and share-of-voice figure. Unbranded buyer-question sets are the right default.

3. Run a no-retrieval control arm. Put the identical question to a frozen model with tools switched off. Retrieval presence and model knowledge are different assets, bought with different budgets, on different clocks: the retrieval layer moves in weeks, the weights move only when the model retrains. No tracker we have reviewed sells a visibility number for a tool-less model, and that is exactly why it is useful as an instrument: it is the clean case where the detector has no citations to lean on, so it exposes what your scoring logic does when the anchor is removed. Every retrieval tool hits that same unanchored case whenever an answer returns thin or no sources. The control arm costs one extra API call per query per day and tells you, in advance, how much of your number is made of prose.

Why a phrase guard does not fix it

A natural patch is a phrase guard: drop a mention when the answer disclaims knowledge or improvises — phrasings such as "don't have information about", "is not widely documented", "appears to be conceptual". It lowers the count without changing what the detector measures: the model produces a phrasing one word outside the guard, and the mention scores again.

You cannot patch a string matcher into an entity detector. Each new phrasing needs a new rule, and the model generates phrasings faster than anyone writes them. The structural alternative is well described: exact match, alias detection and entity disambiguation as separate stages, and entity resolution that binds each mention to a verified knowledge-graph identity rather than to a string. The category's own guidance agrees on the structure of the fix if not the urgency: practitioner write-ups warn about prompt-wording contamination and name collisions, glossaries name template contamination as a false-positive source, and method guides insist on precision and recall against a human-labelled gold set rather than spot checks. One vendor audit of twelve platforms reported 9% of detected mentions as false positives against ground truth.

The standard this holds us to

A number that agrees with you does not get audited, and that is the general shape of measurement failure in this category. A control arm exists so a number is audited whether it flatters its owner or not. We publish how our citation rate is calculated and what our share-of-voice measurement contract is so that every number can be argued with. A measurement nobody can audit is a claim, not a measurement.

FAQ

Does this mean AI visibility tools do not work? No. It means unanchored mention detection does not work, and that is a specific, testable part of a tool rather than a verdict on the category.

Isn't a model with no retrieval the wrong thing to measure AI visibility on? It is the wrong thing to report and the right thing to test with. Nobody should sell a visibility number for a tool-less model. But it is the only clean way to see what your scoring logic does when there are no citations in the answer — and retrieval engines return thin or empty source lists often enough that every tool meets that case in production.

How do I test my own tool for this? Take ten tracked queries that contain your brand name and remove them from your reporting set; see how much your score drops. Then export twenty answers the tool counted as mentions and read them. If any of them is the engine disclaiming knowledge or improvising a description, you have this defect.

Is a hallucinated description of my company a visibility win? It is a real event worth monitoring and it is not presence. If an improvised profile — service lines and positioning generated without a source — scores as a mention, your visibility number rises exactly when the model is guessing.