Benchmark

A benchmark is a standardized test that AI models all take under the same conditions. Just as students are compared by their exam scores, models are compared by having them solve the same set of problems. Every number cited in a "which model is smarter" article traces back to this.

The subjects vary widely. General knowledge (MMLU), PhD-level science (GPQA), competition math (AIME), real-world coding (SWE-bench) — those unfamiliar acronyms attached to the bar charts in new model announcements are all test names.

Two things to watch for when reading these numbers. First, most published scores are "self-reported" — measured by the company under conditions favorable to itself, which can differ from third-party verification. Second, doing well on a test isn't the same as doing well on the job. "Benchmark contamination," where leaked test questions inflate scores, is also a recurring controversy. So it's more accurate to check which test was used and who measured it than to fixate on a few points of difference.

"New model claims to beat rivals across all major benchmarks" — the word "claims" shows up precisely because of the self-reporting problem above. You can find explanations of individual tests in METAL LAB's benchmark section.

1. Search for "LMArena" (Chatbot Arena) and visit it — it's a blind comparison site that shows two AIs' answers side by side with the model names hidden.
2. Ask any question and vote for the better answer. After voting, the models' identities are revealed.
3. Millions of these votes add up to form the "Arena leaderboard" — a hands-on way to see how test scores (benchmarks) and human perception (Arena) can diverge.