Model Labs · Model Benchmarks
Claude Fable 5.2 Edges Gemini 4 Carbon on Google's Benchmark. It Costs More Than Six Times as Much.
The gap is 73.3% to 71.9%, which Google's report treats as noise. Per task: $35.28 against $5.63.
· 3 min read
- GOOGL
- GOOG
Google DeepMind’s Multimodal Knowledge Agent benchmark, 200 office tasks built around blueprints, scans and charts, names Anthropic’s Claude Fable 5.2 as its top scorer, though Google’s launch post never says so.
A buyer picking a model for agents that read such files would assume the top-ranked one is the model to buy. Google’s own statistics say its lead over Carbon could vanish on a different sample of tasks, which turns a sixfold jump in per-task spend into a bet on a ranking.
What does the benchmark test?
The benchmark, shortened to MKA, hands an AI agent a bundle of files such as blueprints, engineering schematics and radiographs and asks for a finished report, slide deck or spreadsheet. Its 200 tasks span 10 industries, and Google said each takes an experienced professional three to eight hours.
Artificial Analysis, which publishes AI model leaderboards, runs the 200-task set and keeps it private. Google published 10 separate sample tasks so outsiders can inspect the format.
How big is the gap?
Each of four models ran once per task; Claude Opus 5.5 scored 70.98% overall and GPT-6 Astra 69.34%.
Google’s headline measure is Climb, the share of the hardest rubric criteria a model meets. Criteria that four earlier models, two of them Gemini, passed reliably sit outside it; the report concedes the split reflects what those models already solve. On Climb, Fable scored 56.56% and Carbon 54.34%, a 2.22-point lead with a 95% confidence interval from minus 0.94 to plus 5.49.
Because that interval includes zero, the report calls the two statistically indistinguishable, and says the same of Opus 5.5, whose gap to Carbon also sits inside its interval. Carbon’s 4.10-point lead over Astra holds up.
What does a task cost?
| Model | Cost per task |
|---|---|
| Claude Fable 5.2 | $35.28 |
| Gemini 4 Carbon | $5.63 |
| Claude Opus 5.5 | $10.29 |
| GPT-6 Astra | $6.03 |
The report figures each cost at list prices. Fable used 5.74 times as many tokens as Carbon, and its bill came to 6.26 times as much.
Artificial Analysis’s own runs show Fable costing more than five times as much, $31.97 per task against $6.29 for Carbon.
Who built the test?
Google DeepMind designed the benchmark and wrote the report, and Carbon is its own model. Google’s researchers chose OpenAI’s GPT-6 Luna as the grader so that a Gemini model would not be favored, and say it shows no bias toward OpenAI either, since OpenAI’s Astra finished last.
Google says anyone can use its 10 public tasks to test their own agents. A buyer can run a similar test on the company’s own files and see whether Fable beats Carbon by enough to justify the price gap.
