When five AI models answer the same question, how often do they actually agree?

We run multio.ai, a product where the same question can go to GPT, Claude, Gemini, Grok and Llama side by side. The thing that surprised us was not that the models differ in style — it is how often they differ in substance on ordinary questions, and how reliably the most interesting claim in a set of answers is the one only a single model made. So we measured it. On 3 November 2026 we sent 24 fixed questions, unchanged, to one model from each of the five families, recorded where the answers materially split, and then hand-checked every claim that only one model made. This page is the full results — method, per-question table, the outlier claims with their verdicts, and the limitations, in full view.

Headline findings

• 52 material divergences were flagged across the 24 questions — a mean of 2.17 per question (median 3).
• 16 of 24 questions (67%) produced at least one material split. The other 8 (33%) were broad agreement: no flagged divergence among the judged answers.
• 15 of 24 questions (63%) contained a lone-model element — a claim only one model made, or a stance only one model took against the rest. Exactly half of the 52 divergences (26) had a lone model: 14 sole-source claims and 12 lone dissents.
• The sole-source claims were right more often than not — but not safely. Hand-checked against the answer texts and outside sources, 8 of the 14 (57%) held up as the highest-value line in their answer set; 3 were wrong or invented; 3 were overstated or mixed. The full claim-by-claim table is below.
• The splits clustered on judgment, not facts. Recommendation, coding and pricing questions averaged 2.5–3.2 flagged splits apiece; factual questions with checkable answers averaged 0.33 — and the only two factual splits were themselves lone-model claims, one of them wrong.
• The same model never disagreed with itself. In the rerun control, two questions were each sent three times to one model (Claude Sonnet 5.5): on both, the verdict was identical in all three fresh samples. Run-to-run variance was wording, not substance — in this study, the splits live between models, not inside one model's sampling.

The practical reading: a synthesis that averages a panel of models deletes the outlier claim — and the outlier claim was the best line in its set 57% of the time, and wrong or invented 3 times out of 14. The disagreement is the information, in both directions.

Method

• Questions: 24 fixed questions written for the study, in four categories of six — pricing/strategy advice, factual-with-a-trap, coding approach, and recommendation-with-tradeoffs. Each question demands a committed position (a pick, a yes/no, a verdict) so disagreement is substantive rather than stylistic.
• Panel: each question was sent unchanged, as a single-turn request, to one pinned model per family through multio.ai’s developer API — OpenAI gpt-5.4-mini, Anthropic claude-sonnet-5.5, Google gemini-3.1-flash-lite, xAI grok-4.3, Meta llama-4-maverick. All five are the mid-tier model of their family, chosen for consistency across families. No panel call fell back to a different model; no answer errored.
• Divergence read: the answers per question were passed to MultiO’s Council, whose judge — claude-sonnet-5.5 in all 24 runs — writes a synthesis and a structured divergence list (claim + per-model positions). The Council endpoint accepts at most four answers, so the judge read a rotating four of the five answers per question; the omitted family rotates by question, so each family was judged in 19–20 of the 24 questions, and all percentages are computed on the judged sets. When the judge’s first pass missed the structured format, a metered extraction follow-up made a second attempt; that repair pass fired in 17 of 24 runs.
• Outlier classification (hand-verified): a sole-source claim is one exactly one judged answer asserts, where the divergence record shows the others did not make it. A lone dissent is one exactly one judged answer takes a stance opposed to every other judged answer that took a stance on the point. Anything else is a multi-way split. The divergence lists were recorded mechanically; the classification — and the verdicts in the next table — were then done by reading every divergence against the answer texts.
• Rerun control: two questions (one factual, one recommendation) were each sent three times to claude-sonnet-5.5 in identical form. The requests were cache-ineligible under the API’s own cache policy and all six samples came back cacheHit: false — genuine fresh samples, identical inputs.
• Cost: the whole programme was metered at $0.87 (866,245 micro-USD) on the study account’s own receipts — panel answers 177,440 micro, Council runs 594,750 micro, rerun control 15,896 micro, plus 78,159 micro for a first attempt whose data was lost to a restart (see Limitations).

The 14 sole-source claims, hand-checked

Each row is a claim exactly one judged answer made. “Highest-value” means the claim was substantively correct when checked and was the most decision-relevant line unique to its answer set. Totals: 8 highest-value, 3 wrong or invented, 3 overstated or mixed — 8 of 14 = 57% held up as the best line in their set.

One composition fact matters for reading this table: 10 of the 14 sole-source claims came from one model (Claude Sonnet 5.5) — and the judge was also Claude Sonnet 5.5. See Limitations.

The 12 lone dissents

Where the splits clustered

On facts the panel converges; on judgment it splits — and the splitting is where the lone claims live.

Which family went lone, and how often

Grok was never the lone model in any judged set — it was always on a shared side of every split. Claude produced the most sole-source claims by a wide margin (with the judge caveat in Limitations), Llama the most lone dissents.

Three outliers worth reading closely

1. The outlier was the product (Q5, Claude). Asked whether an indie app earning $3,000/month from a one-time $30 unlock should switch to $5/month, Gemini said an unconditional Yes (“only ~6 months of retention needed to break even”), GPT and Grok said No unless retention first clears six months. Only Claude priced the transition: “Switching would need ~600 active subscribers to match $3,000/month” — a revenue dip and a backlash risk from existing buyers that a synthesis-to-consensus would have averaged away. The arithmetic is checkable (600 × $5 = $3,000) and it holds.

2. The outlier was the trap (Q9, Gemini). All four judged models correctly called the spiders-in-sleep statistic false. Only Gemini added a provenance: “This was a fabricated statistic created in 1993 to demonstrate how quickly false information spreads on the internet.” That origin story is itself unverified — it traces to a single Snopes page whose cited columnist and magazine have never been shown to exist (Snopes has since conceded both are unverifiable; the Library of Congress has no record of the magazine). The lone claim was the most interesting line in the set and the only one that was wrong.

3. The outlier only one model thought to raise (Q23, Claude). On financing a commercial oven vs keeping domestic ovens, all four judged models said “finance now” on capacity grounds. Only Claude raised the point that could void the whole comparison: “some health inspectors and insurers won’t accept domestic equipment for commercial production.” Jurisdiction-dependent, unverifiable from the armchair — and the first thing a real bakery would need to know.

Per-question results

Div. = divergences flagged by the Council read. “Outlier family” names the family (or families) behind the question’s sole-source claims and lone dissents; a dash means the judged answers either agreed or split with no lone model.

Limitations

These caveats are part of the results, not fine print. Read the numbers with them.

• The judge is also a panellist — and its family produced most of the outliers. The Council judge was claude-sonnet-5.5, which is also one of the five panellists, and it is the model that produced the most sole-source claims (10 of 14). The divergence lists and the sole-source/lone-dissent classification were hand-checked against the answer texts, but a judge favouring its own family’s extra claims cannot be ruled out — the counts should be read with that in mind.
• Rotating judging sets. The Council endpoint caps at four answers, so each question was judged on a rotating four of the five answers — each family sat out 4–5 questions (GPT 5, Claude 5, Gemini 5, Grok 5, Llama 4) and was judged in 19–20. A question classed as “agree” means the judged four agreed; the fifth answer was collected but not judged for that question.
• Mid-tier models, not flagships. The panel is each family’s mid-tier representative (the tier the study account’s free plan allows: small + mid), one step down from the flagship tier. Families and consistency are as designed; the tier is not.
• Memory bleed in Q23–Q24 (verified, disclosed). MultiO’s cross-model memory extracted two “facts” from Q22’s conversation — a question about a hypothetical novelist — and stored them on the study account (novelist; writes long documents daily). On Q23 and Q24, the Gemini and Llama answers absorbed the premise (Q23 Llama: “Given the bakery owner’s profession as a novelist…”; Q24 Gemini opens “For a novelist who is already juggling…”; Q24 Llama likewise). GPT, Claude and Grok answers never used it. Four answer instances were affected; both questions still produced four flagged divergences each, and one of Q23’s sole-source claims is the fabricated premise (it is counted, and marked, in the table above). The two memory facts were deleted after the study and the account’s memory verified empty, so the account is back to its pre-study state.
• Two questions’ panel answers are exact-cache replays. A first run of this study was stopped by a service restart after ~2 questions (its data was lost; its metered cost was 78,159 micro, included in the $0.87 programme total). Its panel calls had populated the product’s exact-match cache, so in the completed run the Q1 and Q2 panel answers were served from that cache (all 10 calls cacheHit: true, charged 0). They are the same models’ own answers to the identical questions, generated minutes earlier; the Council judging on them ran fresh. All other 110 panel answers were generated fresh in the completed run.
• Single run per question. Except for the rerun control, every (question, model) pair was sampled once, at the product’s default sampling settings, with the free plan’s 1,024-token output cap.

The instrument

This study was run end-to-end through multio.ai’s own API and Council view — the same divergence flagging (“Where they split”) that shipped in the product this week, after exactly the observation this study quantifies: synthesis-to-consensus deletes the outlier, and the outlier is often the point. MultiO gives you GPT, Claude, Gemini, Grok and Llama under one subscription, with a per-message receipt at published rates and a Council view that flags where the models split instead of averaging it away. The cost side of running panels like this one is in the model cost calculator; the family comparison is in the GPT vs Claude vs Gemini guide.

Pricing: free trial of 3,000,000 tokens, no card required · Pro $20/month with 80,000,000 tokens · Household $39/month, pooled for up to 5 people · pay-as-you-go top-ups.

Start free — 3,000,000 tokens