---
title: "When Two AI Visibility Tools Disagree, Audit the Same Answer"
description: "A same-answer adjudication test that separates collection differences, scoring-policy choices, and implementation errors when AI visibility tools disagree."
canonical: https://authoritytech.io/blog/ai-visibility-vendor-disagreement-adjudication
last-updated: 2026-09-17
---

# When Two AI Visibility Tools Disagree, Audit the Same Answer

A same-answer adjudication test that separates collection differences, scoring-policy choices, and implementation errors when AI visibility tools disagree.

Canonical URL: https://authoritytech.io/blog/ai-visibility-vendor-disagreement-adjudication
Published: 2026-09-17
Author: authoritytech
Topic: Machine Relations

When two AI visibility tools disagree, first establish whether they scored the same saved answer. Different answers require a collection comparison. Identical answers require a scoring comparison. I would adjudicate that second case with a fixed evidence packet, a written counting rule, and a reviewer who cannot see which vendor produced each label.

Otherwise, you can spend a week arguing about a score without isolating the thing that changed.

This is a proposed diagnostic for marketing operations teams comparing measurement services. It is not a benchmark result, a vendor ranking, or evidence that any named provider computes a metric incorrectly. Its deliverable is a specific decision: **which disagreement needs a collection investigation, which needs a policy decision, and which needs an implementation fix?**

The distinction matters before you replace a tool. A reporting difference can be legitimate. A calculation can be reproducible and still answer the wrong business question. An attractive score does not settle either issue.

## Separate different answers from different scoring

**A same-answer audit holds the observed answer constant so the team can test how the measurement system interprets it.** It does not ask an AI engine to regenerate the answer. Regeneration creates another observation and changes the test.

Start with one disputed report row. Open the underlying evidence from both tools. Compare the saved answer text, citation metadata, collection time, requested prompt, product surface, and collection status. Identical prompt text alone is insufficient: the audit needs the same captured response, including its source objects.

If the tools collected separate responses, stop calling their disagreement a scoring bug. You have not established that yet. Retain the pair for a collection investigation, and build a separate same-input test if the vendors support it.

If both systems can process one saved evidence packet, give them that packet. If neither offers an import or replay function, ask each team to demonstrate its scoring on the same packet or reproduce the published rules in an independent calculation. Label that last option honestly: it tests the documented rules, not the vendors' production code.

This is narrower than my [vendor migration checklist](/blog/ai-visibility-vendor-migration-handover-checklist), which concerns continuity across a tool switch. It also goes beyond [export acceptance](/blog/ai-visibility-evidence-export-acceptance-test), which asks whether you can rebuild a reported rate. Here, the export can be complete and the arithmetic can reconcile while the two systems still label one piece of evidence differently.

| First question | Evidence found | What the team can conclude | Next move |
|---|---|---|---|
| Did both tools observe the same response? | Different answer bodies or source metadata | Input differences exist; their cause remains unproven | Compare collection conditions separately |
| Did both tools receive the same full packet? | Same response, but one export omits citation objects | Evidence delivery differs | Obtain the missing objects before judging the parser |
| Did both use the same counting rule? | Same packet, different written policies | Policy difference, not necessarily a defect | Choose the business definition before choosing a winner |
| Did both follow the agreed rule? | Same packet and rule, different labels | At least one interpretation needs adjudication | Review the disputed fields against the evidence |
| Do row labels match but totals differ? | Same labels, different aggregate | Aggregation requires inspection | Compare grouping, weighting, deduplication, and exclusions |

A useful vendor comparison ends with a cause you can act on. “Tool A says 40 and Tool B says 60” is a starting point.

## Preserve the citation object, not just the answer paragraph

**A source object and a sentence containing a brand name are different evidence.** The test must preserve the objects the original surface supplied, then apply an explicit policy for what counts.

The primary documentation makes this concrete. [Google's Gemini grounding guide](https://ai.google.dev/gemini-api/docs/google-search) describes citation information that links answer text to sources. [Anthropic's web search documentation](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool) describes web-search results and citations carried in its response structure. [Perplexity's Sonar API reference](https://docs.perplexity.ai/api-reference/sonar-post) exposes response fields for citations and search results. These are API contracts, not proof that every consumer interface or monitoring tool exposes the same fields.

My application of those contracts is simple: save the original response structure alongside any rendered view. Do not turn every URL anywhere in a payload into a displayed citation. Do not turn every brand mention into a domain citation. When a role cannot be determined from the captured evidence, label it unresolved.

For a buyer-side audit, I would retain:

- The original response file and a human-readable rendering where available.
- The exact source or citation objects returned with that response.
- The capture method, time, surface, and known model identity. Use an explicit time offset; [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) supplies a timestamp format rather than leaving “9 a.m.” ambiguous.
- The transformation that produced the vendor's normalized rows.
- The counting-rule version and target-domain mapping.
- A stable identifier for the packet and each disputed item inside it.

[W3C PROV](https://www.w3.org/TR/prov-overview/) provides a useful provenance model through entities, activities, and agents. Applied here, the saved response is an entity, the extraction is an activity, and the responsible system or reviewer is an agent. That mapping is my proposed audit design, not a W3C certification of visibility software.

Preserve raw bytes before normalizing them. If a team also hashes canonical JSON, [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785) describes a canonicalization scheme that supports repeatable hashing. Record which representation was hashed. Matching hashes establish matching input under the chosen method; they do not prove that an answer is true or its citation supports the claim.

Check the file before blaming the scoring. [RFC 8259](https://www.rfc-editor.org/rfc/rfc8259) warns that JSON objects with non-unique member names have unpredictable receiver behavior. If a malformed packet repeats the same field name with different values, two readers can disagree before either applies a visibility rule. Preserve that packet as an input defect; do not silently clean it and call the result an identical-input test.

## Build a small adjudication fixture with known edge cases

**The fixture should expose specific classification errors rather than imitate a market-wide benchmark.** A tiny diagnostic set can identify a reproducible defect. It cannot establish a vendor's overall accuracy.

I would begin with the following six illustrative cases. They are hypothetical records, not observed customer results. Use evidence you are authorized to share; remove confidential prompts and sensitive answer content before involving an external vendor.

For this example, the agreed metric is the share of completed eligible answers with at least one explicit source citation to the declared target domain. Brand mentions alone do not count. Repeated target links count once per answer. Failed collections remain separately reported and outside this particular denominator. A completed answer with no citations stays inside it.

| Fixture | Saved evidence | Expected label under this rule | Failure the fixture can expose |
|---|---|---|---|
| F1 | Answer names the target brand, but supplies no target citation | Eligible; target not cited | Mention treated as citation |
| F2 | Two explicit citation links to the target domain in one answer | Eligible; target cited once | URL count substituted for answer count |
| F3 | Explicit citation to a namesake company's different domain | Eligible; target not cited | Brand-string matching overrides entity identity |
| F4 | Target URL appears in a retrieved-results field, not as an answer citation | Eligible; target not cited | Retrieval treated as citation |
| F5 | Completed answer with no source links | Eligible; target not cited | Uncited answer silently removed |
| F6 | Collection fails before an answer is captured | Failed; excluded from this rate and reported separately | Failure silently treated as a completed negative |

Under the stated rule, the fixture produces one target-citing answer among five eligible answers: **1/5, or 20 percent**. That is arithmetic on a designed example. It says nothing about any brand's actual visibility.

Now change one thing at a time. If a tool counts F1, inspect mention-to-citation classification. If it counts F2 twice in an answer-level numerator, inspect deduplication. If it drops F5, inspect denominator construction. If it counts F3, inspect entity matching.

F4 needs particular care. Some surfaces do not expose enough information to distinguish retrieval from citation. Do not manufacture an expected label from missing metadata. Replace that case with a sufficiently documented packet, or record that the distinction cannot be adjudicated from this evidence.

These expected labels belong to the declared rule. A tool offering a separate brand-mention metric can legitimately count F1. The defect would be presenting that count as compliance with this citation rule, not offering the other metric.

## Use a blind reviewer to resolve labels before totals

**Adjudicate the disputed observation before comparing dashboard percentages.** Otherwise, a preferred commercial outcome can become the rule by accident.

I would remove vendor names from the proposed labels and give a reviewer three things: the evidence packet, the counting rule, and the competing classifications. The reviewer records the decision and the exact evidence location supporting it. Where practical, use someone who did not implement either extraction.

For ambiguous cases, a second reviewer should make an independent judgment. Disagreement between reviewers is information. It can mean the policy is underspecified or the packet is incomplete. It does not automatically mean either vendor is wrong.

This separation also fits [NIST AI RMF Core](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/), which addresses documented responsibilities and incorporates adjudicated feedback into improvement. The framework does not prescribe this fixture. I would apply its accountability principle by naming the policy owner separately from the person implementing the parser.

Use a compact decision record:

| Field | What belongs in it |
|---|---|
| Fixture and packet identity | The specific saved response and disputed source object |
| Policy version | The rule in force before labels were compared |
| Competing labels | Included, excluded, matched, unmatched, or unresolved |
| Adjudicated label | The selected classification, or unresolved when evidence is insufficient |
| Evidence pointer | The relevant response field, citation object, or source-domain mapping |
| Reason | A short explanation tied to the rule, not vendor reputation |
| Remedy owner | Collection team, measurement-policy owner, or implementation owner |
| Retest condition | The exact case that must produce the corrected result |

Keep the metric definition attached to every adjudication. The [W3C Data Quality Vocabulary](https://www.w3.org/TR/vocab-dqv/) distinguishes a quality metric from a measurement made using that metric. In this audit, the written citation rule and its resulting label should likewise remain separate objects. Changing the rule changes what the label means.

Freeze the labels before rebuilding the rate. If an expected label changes, retain the previous version and the reason. Do not quietly rewrite the answer key until the preferred vendor passes.

This is a proposed control, not a claim that blind review eliminates bias. Its value is inspectability: another reviewer can disagree with a recorded decision instead of trying to reconstruct a meeting.

## Separate policy choices from implementation defects

**Two systems can disagree while each follows its own documented policy correctly.** The buyer still needs to decide which policy answers the intended question.

Consider the fixture's uncited completed answer, F5. Under the rule above, it enters the denominator. A different metric could condition on answers containing at least one citation. That metric would exclude F5 by definition. Neither denominator is made universally correct by the word “visibility.”

The question is whether the company needs the chance of being cited across eligible answers or its presence within the subset of answers that cite something. Keep separate names for separate measurements. My [methodology audit](/blog/ai-visibility-scores-not-comparable-platform-methodology-audit) explains why a shared score label is insufficient for comparison.

I would close each dispute with one of four outcomes:

1. **Collection difference.** The underlying observations differ. Preserve both; investigate surface, timing, prompt, access method, and response variation without claiming a parser defect.
2. **Policy difference.** Identical evidence receives different labels under different written rules. Select or retain separately named metrics; do not demand artificial agreement.
3. **Implementation defect.** A label or aggregate violates the agreed rule on a sufficiently documented fixture. Assign the exact failing case for correction and retest.
4. **Unresolved evidence.** The packet or rule cannot support adjudication. Quarantine the disputed claim, disclose the limitation, and obtain the missing evidence.

For an implementation fix, passing the original failing fixture is necessary but insufficient. Add nearby counterexamples to test whether the repair creates a new error. A namesake fix, for example, should reject the wrong company without rejecting legitimate aliases of the target company.

After adjudication, a technical reviewer can summarize expected versus produced labels in a confusion matrix. The [scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) defines its rows as true classes and columns as predicted classes. Here, use adjudicated labels as the reference, preserve unresolved cases outside the binary table, and keep false positives separate from false negatives. Report counts on the fixture, not an accuracy estimate for the wider market.

For a policy change, recompute the affected historical comparison or mark a measurement break. Do not present a newly defined denominator as an improvement in the brand. [W3C Data on the Web Best Practices](https://www.w3.org/TR/dwbp/) recommends version indicators and version history. Apply that discipline to the answer key and counting policy: attach the old version, the new version, and the specific classifications that changed to the repair record.

## What the Machine Relations Index contributes to this test

**The Machine Relations Index provides a public example of versioned citation evidence, not a verdict on competing tools.** Its [release manifest](https://machinerelations.ai/data/mri-release-manifest.json), checked directly on September 17, 2026, identifies release `mri_score_v2.0+2026-09-17+d5e23024fc20`, a May 10 through September 17 window, 124 observed dates, and six healthy answer engines. It also identifies the public artifact by checksum.

That is the habit to borrow: identify the data release before debating what a result means. The [MRI public index](https://machinerelations.ai/index) is a data authority for its stated monitored corpus. It does not establish that this fixture represents the market, that a vendor's private parser is correct, or that a reporting change caused a commercial result.

Within [Machine Relations](https://machinerelations.ai/glossary/machine-relations), I would use this adjudication before assigning visibility work to the wrong team. A collection defect needs measurement repair. A source that does not support the answer needs claim-level review. A real absence from relevant answers calls for investigating the brand's evidence and [citation architecture](https://machinerelations.ai/glossary/citation-architecture).

Those are different jobs. Sending all three to a content writer turns uncertainty into more pages.

An [AuthorityTech visibility audit](https://app.authoritytech.io/visibility-audit) is a starting point for examining the underlying discovery problem. The same-answer test has a narrower purpose: keep a tool disagreement from masquerading as that problem.

## FAQ: resolving AI visibility tool disagreements

### Why do AI visibility tools disagree on the same answer?

Possible explanations include different source-role definitions, brand or domain matching, repeated-link handling, eligibility rules, or aggregation. A same-answer fixture isolates those interpretations from differences in the answer collected. The cause remains unproven until the disputed evidence and rules are compared.

### Does rerunning the same prompt create a same-answer test?

No. This test requires the saved response and its evidence objects. A new run is another observation. It belongs in a collection or repeatability comparison, not a claim that two parsers received identical input.

### What if the vendor cannot replay a saved answer?

Ask for a demonstration on the packet or independently apply its documented rules. State the limit: reproducing documentation is not verification of the production implementation. If evidence remains unavailable, leave the disputed classification unresolved rather than inferring a hidden capability is absent.

### Does a passing fixture prove a tool is accurate?

No. It proves that the tested behavior matched the rule on those cases. Broader performance requires a separate, appropriately sampled evaluation. A diagnostic fixture should never be advertised as a market benchmark.

### Should the higher AI visibility score win the vendor comparison?

Not on that basis. First establish what each score measures and whether the implementation follows that definition. Then choose the measurement that serves the business decision. A larger number is not evidence of a better instrument.

Start with one disputed answer. Hold it still. Find the first rule or evidence object where the systems diverge. That is where the repair belongs.

## Links

- [Blog Index](https://authoritytech.io/blog.md)
- [Home](https://authoritytech.io/index.md)