The Literature Disagrees. Your AI Won't Tell You

Two studies. Same drug, same question, opposite conclusions.

One found the effect. One didn't. Both were peer-reviewed, both were competently done, and neither was wrong. The populations differed. So did the dosing, the endpoints, the follow-up.

A researcher reading these two papers side by side doesn't panic. She reads the methods, works out where they diverged, and holds both results in view while she decides which one speaks to her question. The disagreement isn't a problem to be resolved before she can proceed. It is the finding, or at least part of it.

This is what the biomedical literature actually looks like. Not a settled body of knowledge waiting to be retrieved, but an ongoing argument, with the reasons for the argument recorded in the methods sections.

What the Machine Does Instead

Ask it about the same drug, and it will answer. One answer, stated cleanly. Somewhere behind that sentence sat two studies pointing in opposite directions, but the system has already picked one, and it doesn't mention the other.

This isn't hypothetical. A team working on biomedical question answering ran a controlled test : they gave models a clinical question with two retrieved documents, one supporting the correct answer and one contradicting it, then ran it again with the same two documents in reverse order. Nothing else changed. Accuracy dropped for every model tested, and up to a quarter of predictions flipped simply because of which document came first.

The obvious response is that this is an early-days problem, the kind of thing a larger model or a better prompt will eventually fix. That's been tested too. A separate study of medical models found that even state-of-the-art, high-accuracy systems often fail to hold back when uncertain, and that scaling model size or improving prompting brought little improvement. Adding an explicit option to abstain helped. Making the model bigger did not.

The problem isn't that these systems can't recognise a contradiction. It's that nothing in how they're built requires them to report one. A confident answer reads as a better answer, so the disagreement it was built on disappears somewhere between retrieval and the sentence you read.

So the researcher who spent an afternoon working out why two trials disagreed, and the system that resolved the same disagreement in a fraction of a second without telling her, have not done the same job. Only one of them knows there was a question.

Catching It From the Outside

You don't need to see inside a model to know when to trust it. You can watch how it behaves, repeatedly. Run the same question through the same agent ten times and it won't take the same path. One study found identical inputs produced between two and four distinct action sequences across ten runs, and that the variation predicts failure: consistent runs landed in the mid-eighties for accuracy, while runs that took four or more different paths dropped into the forties. Declining to answer when independent runs disagreed outperformed answering every time.

Why One Answer Isn't Enough

The shift from a single model to an agentic workflow is something we covered last week: instead of answering in one pass, the system searches, plans, and works through a question in stages. What matters here is what that structure makes possible. Because the work is split across steps, different parts of the process can be assigned to look for different things. One agent runs the literature search. Another gathers what supports a given answer. Another checks each claim against the source it came from. And, importantly, another can be pointed at what doesn't fit: the study with the opposite result, the population that wasn't included, the endpoint that was measured differently.

This is the part a single pass can't do. A model generating one response has no structural reason to hold a contradiction open; it resolves everything into the sentence it returns. A workflow can be built so that disagreement is something a dedicated step is actively looking for, rather than something that gets smoothed away on the way to an answer.

But structure alone doesn't guarantee it. Adding agents can just as easily bury the problem deeper. If every agent inherits the first conclusion reached, whether because they were shown each other's output along the way or because they were all built to converge on the same kind of answer, the disagreement still disappears. It just disappears one step later, behind more machinery instead of less.

Why Independence Is Not Optional

There is an obvious way this goes wrong in practice. If the agents can see each other's work, agreement stops meaning anything. Once one has committed to an answer, the others are no longer reaching a conclusion of their own. They are responding to a conclusion already on the table, and their convergence is an echo that sounds exactly like confirmation.

The research on multi-agent calibration makes the point in both directions. Harmless variation between agents can create doubt about an answer that was fine. But agreement reached through communication rather than through independent work produces the more dangerous error, which is confidence in a conclusion only one agent ever really reached.

It is the same reason a systematic review has two people screen independently before they compare notes. Not because two opinions beat one, but because two opinions formed separately are the only kind worth comparing.

What This Looks Like in Practice

This is the principle Clarisyn is built on. Several agents work the same question in parallel, none of them seeing what the others are finding, and only afterwards are their outputs compared. Where they converge, that convergence carries weight, because it was reached separately. Where they diverge, the system does not quietly pick a winner.

Ask about the metabolism of a compound and one agent may return CYP1A2 while another returns CYP3A4. The tempting move is to settle this before the researcher ever sees it. But that contradiction exists in the literature, not in the system, and a researcher working on that compound needs to know it is there. Hiding it produces a tidier answer and a worse one. So the split is surfaced instead, with the reasoning behind each position visible and every claim traceable to the paper it came from. What comes back is not a verdict but a position: here is what the evidence supports, here is where it divides, here is what you would need to read to decide.

None of this means the box has been opened. We are working with the same models everyone else is, and we cannot see inside them either. What we can do is refuse to let a single pass through one of them stand in for an answer.

The goal was never to manufacture agreement. It is to make uncertainty visible.