How to Read AI Leaderboards Without Getting Fooled

In this guide

1. The three kinds of scoreboard
2. How to read an Arena score
3. How to read an index score
4. How to read a lab's own benchmark table
5. A five-question checklist for any AI ranking
6. Worked example: reading Carbon's debut
7. Common mistakes
8. What this means for you

Every AI launch now comes with a scoreboard. "#1 on LMArena." "Tops the Intelligence Index." "State of the art on 9 of 12 benchmarks." Those claims are where most buying decisions start, and they're easy to misread.

This week was a good example. On October 27, Artificial Analysis rebuilt its Intelligence Index, and DeepSeek V4.1 Flash went from 45 to 32 overnight. Nothing about the model changed; the test did. A day later, Google's Gemini 4 Carbon debuted at #1 on two leaderboards, but almost nobody can use it yet.

Here's how to read the numbers so they help you pick a model instead of steering you wrong.

The three kinds of scoreboard

Nearly every AI ranking you'll see falls into one of three types. They measure different things, and each has a different weak spot.

None of these is "the real one." We use all three in our methodology, with crowd votes and independent tests carrying the most weight and lab tables treated with the most suspicion.

How to read an Arena score

LMArena shows people two anonymous answers to the same prompt and asks which is better. Millions of votes become a rating on the same kind of scale chess uses. The jargon guide covers the basics. What matters here is what a gap actually means.

On that scale, a points gap translates into how often one model is expected to win a head-to-head vote:

So when Gemini 4 Carbon debuted at 1528 on Arena text, against 1520 for Gemini 4 Argon and 1509 for Claude Opus 5.5, it was "#1", but voters would pick it over Opus 5.5 only around 53 times in 100. That's a real lead, but a small one, and a new model's score moves as votes pile up.

Two more things to check on any Arena board:

• Vote count. Microsoft's MAI Image 2.7 entered Arena's image-editing board with about 3,400 votes, against tens of thousands or more for the models around it. Treat a ranking built on a few thousand votes as provisional.
• Which board. Opus 5.5 is #3 on Arena text but #1 on Arena WebDev, the coding board. "#1 on Arena" means little until you know which arena.

How to read an index score

Artificial Analysis runs a set of hard tests on every model itself and combines them into one number. Because it runs the tests, the labs can't choose which results get published, which is why we lean on it.

The catch is that the number only means something within one version of the test. Version 5, launched October 27, changed the mix: about 60% of it now uses private tests that models can't have seen during training, and there are new coding and science tasks. The effect was large:

• DeepSeek V4.1 Flash and GLM-5.3 both fell from 45 to 32.
• Meta's Muse Wonder Full fell from 56 to 50; Claude Sonnet 5.5 from 56 to 54.
• OpenAI's models gained, with GPT-6 Astra at 57, a point behind Opus 5.5.

The tempting headline is "open-weight models got worse." They didn't. The new test is harder to prepare for, and it shows a bigger gap between open and closed models than the old one did. As one analysis put it, scores from different versions aren't suitable for long-term comparisons. If you see a chart that plots a model's score over time, check whether the method changed partway through.

How to read a lab's own benchmark table

Launch charts aren't lies, but they are marketing. The lab picks which benchmarks to show, which rivals to compare against, and the settings, often the most expensive "max effort" mode. Three habits help:

• Look for an independent re-run. When Anthropic launched Claude Haiku 5.5, its own table showed 39.2% on Terminal-Bench 4.0. Independent runs found 32.8% (Artificial Analysis) and 35.4% (Vals AI). That's still strong, but lower. Our Haiku vs Luna comparison has the details.
• Read the footnotes. Google says Gemini 4 Carbon's cybersecurity scores were run without guardrails, which is not the version customers will get.
• Watch for "matches." OpenAI says GPT-6.1 Terra matches GPT-6 Astra at a tenth of the price. Independent scores put Terra clearly below even GPT-6.1 Sol. "Matches on one benchmark" and "matches overall" are very different claims.

A five-question checklist for any AI ranking

Before you act on a "#1" claim, run through these:

1. Who ran the test? A third party, a crowd or the lab itself? The further from the lab, the more weight it deserves.
2. What does it measure, and is that your job? A coding board tells you little about writing emails. Find the board closest to what you'll actually do.
3. How big is the lead? A few Arena points or one index point is close to a tie. Look at the gap before the rank.
4. How much evidence is there? Count the votes and check the date. A week-old debut is provisional; a model with months of votes is settled.
5. Can you use it, and at what cost? Carbon tops both boards, but at launch only selected security teams in Google's Fairwind Program can access it. A model you can't buy can't do your work. Check the price, and the cost per task if it's listed.

Worked example: reading Carbon's debut

Here's the checklist applied to this week's biggest claim, "Gemini 4 Carbon is the new #1":

• Who ran it? Arena (crowd) and Artificial Analysis (independent), so this is stronger than a lab-only claim.
• What does it measure? General text and reasoning. On WebDev, Carbon debuted at #4, behind Opus 5.5.
• How big a lead? 8 Arena points over Argon and 19 over Opus; one index point over Opus.
• How much evidence? A day of votes, so expect movement.
• Can you use it? Not yet. Google has given no public API or app date.

Our verdict: Carbon is a genuine contender, which is why it's #2 in this week's standings, but Opus 5.5 keeps #1 because you can use it today. See the Carbon vs Opus 5.5 match-up for the side-by-side.

Common mistakes

• Treating rank as a gap. #1 and #3 can be separated by less than the margin of error.
• Comparing across versions. A model that "dropped 13 points" may simply have been re-tested on a harder exam.
• Mixing boards. Text, coding, image and video leaderboards rank different skills. A text #1 can be a coding #4.
• Ignoring effort settings. Many top scores use a model's most expensive mode. The default setting you'll actually run may score lower.
• Forgetting availability. Previews, waitlists and region limits are common. Check that you can actually buy a model before you plan around it.

What this means for you

Use leaderboards to build a shortlist, not to make the final call. Pick two or three models that score well on the board closest to your task, then try them on your own work. Our guide to fact-checking AI answers has a quick routine for judging the results. Your own prompts are the benchmark that matters most.

Free · every Friday

Get the standings in your inbox

Who moved up, who fell, and what's worth trying this week, in a five-minute read. No spam.

Email address Subscribe