Web Search Doesn't Make AI Answers More Trustworthy

AI products that search the web while answering charge a premium for the privilege. We wanted to see whether models that retrieved more text actually drew fewer disputes from competitors auditing their answers.

On a five-question Verbatim Index run, only four of the nine source models produced usable retrieval telemetry, and inside that four-model comparison the ranking on retrieval volume matched the ranking on cross-vendor peer disputes for the top pair and inverted on the bottom pair.

Five questions per model is small. Each per-model number below is a five-observation mean, and small differences belong inside per-question noise. Retrieval tokens are search-results text returned into the model's context; retrieval share is retrieval tokens over input plus output tokens for that question. Peer dispute rate is summary.by_source.<model>.cross_family.error_rate_pct: the share of cross-vendor critique sessions in which a peer from a different vendor family flagged at least one issue with the answer. Lower means fewer flags, not lower factual error. Five of the nine source models (Gemini 3 Flash, Gemini 3.1 Pro, Grok 4.3, Grok 4.20, Sonar-pro) logged zero retrieval tokens the runner could capture; since Sonar-pro is search-native and the Gemini and Grok models ship with vendor search tools, those zeros most likely reflect uninstrumented native search. The numbers below are the four models the runner did measure, ranked by retrieval share. GPT-5.5: 92.08% retrieval, 23.54% peer dispute, the lowest in the four. GPT-5.4: 88.96% retrieval, 26.67% dispute. Claude Sonnet 4.6: 79.17% retrieval, 32.75% dispute, the highest of the four. Claude Opus 4.7 retrieved on one question of five (median retrieval tokens: zero; 20,264 on q-005; mean 11.54%) and posted 30.99% dispute, third of the four.

The structural read is what dispute rate measures and what it does not. A peer flagging an answer means the answer made a falsifiable claim that another model disagreed with; it does not mean the claim was wrong. Retrieval volume can move dispute rate in either direction, because the dispute count depends on whether retrieved material was correctly synthesized and whether peers from other vendor families agreed with the synthesis. The four-model picture shows both directions in the same dataset: at the top, the heaviest retriever (GPT-5.5) also had the fewest peer flags; in the middle, the third-heaviest retriever (Sonnet 4.6) had the most flags; at the bottom, Opus 4.7 retrieved almost nothing and ranked between the two. The strongest counterargument is that the GPT-5.5 / GPT-5.4 alignment at the top is the real finding and the post is overweighting two Anthropic data points. That cuts both ways. GPT-5.5 and GPT-5.4 are siblings, so the top-pair alignment is itself a same-family observation, and the within-Anthropic pair (Sonnet 4.6 versus Opus 4.7) inverts the retrieval-volume-to-dispute-rate ordering on the same metric. The data does not contain a clean cross-family signal that retrieval volume predicts peer dispute rate in either direction.

For someone choosing which AI answer to act on, treat retrieval volume as a property of the response, not as a tiebreaker. A high-retrieval answer is not by itself a reason to trust the response more than a low-retrieval one; in this run GPT-5.5's high-retrieval answers were flagged at 23.54% by cross-family peers and Sonnet 4.6's similarly high-retrieval answers were flagged at 32.75%. The useful second opinion on a research-style answer is a model from a different vendor family naming the specific claims it would dispute, with text you can check, because that produces a list a human can rule on rather than a single retrieval indicator.

Verbatim runs this kind of review on your actual AI responses, in place, as you work. Try it free →