Before the tests
How a benchmark turns into a number
A benchmark is a fixed set of tasks and a rule for marking them. Run a model on every task, mark each one, and average: that's the number in the table. But the model never takes the test alone. A harness decides what tools it gets and how long it can try, and a grader decides what counts as right. Both are choices, and both move the score.
Each figure below shows one item going through a real benchmark, from the task to the grader's verdict. In every figure, orange is what the model wrote, and the grader's marks are green or red.
- 1
Dataset
A frozen set of tasks with answers or tests kept out of sight.
66 tasks, tests hidden
- 2
Harness
The loop around the model: its tools, time limit, retries and prompt.
a shell, 8 hours, 5 runs
- 3
Model
The only part a launch table names.
Opus · GPT · Gemini · Grok…
- 4
Grader
Decides each item: tests, exact match, a rubric, or another model.
hidden tests
- 5
Score
One number per benchmark, averaged over items (and often runs).
% of tasks resolved
Coding · 2 benchmarks
Write the code, pass the hidden tests
The model gets a real environment and a task in plain words. It works until it says it is done; then tests it never saw decide.
Reported by 8 of 8
Terminal-Bench
Can the model finish hard, hours-long jobs in a real Linux terminal?
- Made by
- Stanford & Laude Institute, 2025
- Size
- 66 tasks (v4.0)
- Graded by
- Hidden tests
- Headline score
- % of tasks resolved
Each task is a Docker container, an instruction file and a set of tests the model never sees. The model gets a shell and works: installs packages, writes code, runs it, reads errors. It can take hours; the limit is up to eight. When it says it is done, the hidden tests run against whatever state it left behind.
A task counts only if every test passes. The headline number is the share of tasks resolved, and the public leaderboard runs each task five times because a single run is noisy.
Version 4.0 (August 2026) has 66 tasks. Eight of version 3.0's 74 were removed: two were too easy, two drew refusals, two had solutions posted online, and two had quality problems.
instruction.md
Build fast_networkx, a faster drop-in subset of NetworkX focused on VF2++ graph isomorphism. import fast_networkx as fnx must work from /app.
agent stopped · hidden tests run
Before comparing scores
- Versions don't compare. All eight launches print "Terminal-Bench", but Kimi K3 reports 2.1 and GLM-5.3 reports 3.0, older and easier versions than everyone else's 4.0.
- The agent harness around the model (Terminus, Claude Code, Codex…) and its effort setting move the score by several points. The same model appears with three different 4.0 scores in three labs' tables.
- Meta's "-AA" row is a separate run by Artificial Analysis, not Meta's own.
- With 66 tasks, one task is about 1.5 points. Differences of two or three points are inside the noise.
Field Guide benchmark · ai agent · tool use
Sources Terminal-Bench 4.0 announcement · Terminal-Bench repository (Harbor) · Terminal-Bench paper (v2.0, arXiv 2601.11868)
As printed Opus 5.5: “Terminal-Bench 4.0” · GPT-6 Astra: “Terminal-Bench 4.0” · Gemini 4 Argon: “Terminal-Bench 4.0” · Muse Wonder: “Terminal-Bench 4.0-AA” · Grok 4.7: “Terminal-Bench 4.0” · Mistral Large 4: “Terminal-Bench 4.0” · GLM-5.3: “Terminal Bench 3.0” · Kimi K3: “Terminal-Bench 2.1”
Reported by 7 of 8
DeepSWE
Can the model ship a real feature or fix in a real codebase, the way a teammate would?
- Made by
- Datacurve, 2026
- Size
- 113 tasks, 91 repos
- Graded by
- Hidden + regression tests
- Headline score
- pass@1
DeepSWE is the coding benchmark that replaced SWE-bench in launch tables. Each task is a short, human-sounding request ("fix the label sorting") and a real open-source repository in TypeScript, Go, Python, JavaScript or Rust. The model gets bash and nothing else, and writes a patch.
Two test suites grade it. New tests, written for the task and hidden from the model, check the behaviour through the project's public API. Then the repository's own test suite runs, so a fix that breaks something else fails.
The tasks were written from scratch for the benchmark, so the answer can't already be in a model's training data. They are big: the reference solutions average about 668 lines across seven files. The score is pass@1: solved on the first attempt.
request
Fix PromQL label sorting across typed and untyped values
new tests · hidden from the model
12 / 12 passed
checks the new behaviour through the public API
the repo's own test suite
1,284 / 1,284 passed
proves the fix didn't break anything else
Before comparing scores
- Labs mix their own runs with numbers from Datacurve's leaderboard, and the top models sit in a narrow 74–79 band where harness choices matter more than the gap.
- Version 1.1 (September 2026) changed tasks from v1. Check which one a table means.
Field Guide benchmark · test set · data leakage
Sources DeepSWE (Datacurve)
As printed GPT-6 Astra: “DeepSWE v1.1” · Gemini 4 Argon: “DeepSWE v1.1” · Muse Wonder: “DeepSWE 1.1” · Grok 4.7: “DeepSWE v1.1” · Mistral Large 4: “DeepSWE v1.1” · GLM-5.3: “DeepSWE” · Kimi K3: “DeepSWE”
Computer work · 3 benchmarks
Use the software, change the state
No code to grade here. The model clicks, calls APIs and edits files, and the grader inspects what the world looks like afterwards.
Reported by 7 of 8
AutomationBench
Can the model run a business workflow across many apps without breaking the rules?
- Made by
- Zapier, 2026
- Size
- 600 public tasks + private set
- Graded by
- Final-state check
- Headline score
- % of tasks passed
The model works inside simulated versions of everyday SaaS apps (calendars, spreadsheets, Slack, CRMs: 47 tools and about 500 API endpoints) and is given a workflow to finish. The data contains decoys: an outdated policy sheet, a similarly named channel.
Nobody reads the model's reasoning. When it stops, a program inspects the state of every app. Each task has must-pass objectives (the right event was renamed, the summary was posted) and must-not-happen guardrails (nothing deleted, nothing sent to the wrong place). Miss an objective or trip a guardrail and the task fails.
request
There's a scheduling conflict on February 20, 2026 at 2:00 PM — a Zoom meeting and a Google Calendar event overlap. Check the meeting priority policy in the spreadsheet to determine which one wins, then reschedule the loser by prepending [RESCHEDULED] to its topic/title. Post a summary to #ops-updates on Slack…
what the model did
what the grader checks after
Before comparing scores
- Three different things share the name: the 600-task public set, Zapier's private set, and Artificial Analysis's 657-task "-AA" version, which scores the share of objectives met and zeroes any task with a guardrail violation.
- Anthropic's numbers come from Zapier's own runs, where a safety refusal counts as a failure.
Field Guide ai agent · tool use
Sources AutomationBench paper (arXiv 2604.18934) · Zapier benchmarks
As printed Opus 5.5: “AutomationBench (run by Zapier)” · GPT-6 Astra: “AutomationBench” · Gemini 4 Argon: “AutomationBench (private set)” · Muse Wonder: “AutomationBench-AA” · Mistral Large 4: “AutomationBench (AA, 657 tasks)” · GLM-5.3: “AutomationBench” · Kimi K3: “AutomationBench (600-task public set)”
Reported by 5 of 8
OSWorld 2
Can the model use a real desktop, across many apps, for an hour and a half of work?
- Made by
- XLANG Lab (HKU), 2026
- Size
- 108 tasks
- Graded by
- ~27 checkpoints per task
- Headline score
- Partial or binary %
The model sees screenshots of a virtual machine and controls it with mouse and keyboard. Tasks take a person a median of about 1.6 hours and an agent around 300 actions. They span apps (a PDF, a web portal, email, a bank site), and some change halfway: a new email arrives with information the model has to act on.
Each task has about 27 checkpoints. The "partial" score is the average share of checkpoints passed; the "binary" score only counts tasks where every checkpoint passed.
OSWorld 2 replaced the original OSWorld-Verified, which top models had pushed past 80%.
task
File last month's expense claim in the company portal. Follow the steps in the tutorial PDF, read the amounts off the receipts, and check them against Gmail and the bank statement.
actions taken 312
checkpoints
partial score
81%
22 of 27 checkpoints. Most launch tables print this one.
binary score
0%
Five checkpoints missed, so the task wasn't done.
Before comparing scores
- Partial and binary are very different numbers. Meta reports both for Muse Wonder Full: 78.4 partial, 39.4 binary. Most labs print only partial.
- Labs pick online or offline subsets and different releases (2.0, the August patch that fixed reward hacking in 14 tasks, 2.1).
- Kimi K3 scores 84.8 on the old OSWorld-Verified and 58.3 on OSWorld 2.0. Same model, newer test.
Field Guide ai agent · computer vision
Sources OSWorld 2 repository and paper (arXiv 2606.29537)
As printed Opus 5.5: “OSWorld 2.1 (partial)” · GPT-6 Astra: “OSWorld 2.0 (offline, partial)” · Gemini 4 Argon: “OSWorld-2.0 (offline, partial)” · Muse Wonder: “OSWorld V2.1 (partial and binary)” · Kimi K3: “OSWorld 2.0”
Reported by 4 of 8
Agents' Last Exam
Can the model do a professional's computer work in the software professionals use?
- Made by
- Berkeley RDI + 250 experts, 2026
- Size
- 1,490 tasks (152 public)
- Graded by
- Compare to expert output
- Headline score
- Full-pass %
Experts from 13 industries wrote tasks in the real tools of their trade, from video compositing to CAD. The model produces files, and each file is compared against an expert's reference output or rubric, field by field, with tolerances where numbers are involved.
A task passes only if the whole output passes. Tasks come in three tiers; on the hardest one, models average under 1%. There is also a terminal-only subset, ALE-CLI.
task
Pull the headline figures out of this 10-K filing into extraction.json.
Before comparing scores
- The pool keeps growing and has tiers and a CLI subset, and launch posts rarely say which slice they ran. GLM-5.3 labels its row "(CLI)"; the four labs' numbers range from 28 to 59 and almost certainly aren't measuring the same thing.
Field Guide ai agent · tool use
Sources Agents' Last Exam · ALE paper (arXiv 2606.05405)
As printed GPT-6 Astra: “Agents' Last Exam” · Gemini 4 Argon: “Agent's Last Exam” · GLM-5.3: “Agents' Last Exam (CLI)” · Kimi K3: “Agents' Last Exam”
Professional work · 5 benchmarks
Produce the deliverable, face the rubric
A memo, a model, a clinical answer. There is no single right string, so experts write checklists, and a judge (often another model) marks against them.
Reported by 5 of 8
Harvey's Legal Agent Benchmark
Can the model produce legal work a partner would accept for review?
- Made by
- Harvey, 2026
- Size
- 1,200+ tasks, 75,000+ criteria
- Graded by
- Rubric, model judge
- Headline score
- All-criteria pass %
Each task reads like a partner's instruction to an associate, plus a closed set of client documents: a data room, contracts, correspondence. The model writes the memo or draft.
Lawyers wrote a rubric for every task, more than 75,000 criteria in all. A judge model checks the output against each one. The strict score counts a task only if every criterion passes, which is why all-pass scores sit between roughly 4% and 20%.
partner's instruction + data room (37 documents)
Review the data room, identify change-of-control provisions, assess the risk to the deal, recommend next steps, and prepare a review-ready draft memorandum for the deal team and Board.
draft memorandum
DRAFT · Privileged. Section 14.2 of the Supply Agreement treats the merger as a change of control requiring the supplier's written consent on 30 days' notice; absent consent the supplier may terminate. We recommend…
per criterion
88%
7 of 8 criteria met. Kimi K3's 94.6 is this kind of number.
all-pass (LAB's score)
0
One criterion missed fails the task. Top all-pass scores are roughly 4–20%.
Before comparing scores
- Kimi K3 reports 94.6, which is the share of individual criteria met, not tasks fully passed. The two numbers can't be compared.
- LAB-AA is Artificial Analysis's private 120-task version with a single judge model; Vals runs another. The score depends on the judge.
Field Guide benchmark · ai agent
Sources Introducing Harvey's Legal Agent Benchmark · harvey-labs repository
As printed Gemini 4 Argon: “Harvey's Legal Agent Benchmark” · Muse Wonder: “Harvey LAB-AA” · Grok 4.7: “Harvey Legal Agent Benchmark” · Mistral Large 4: “Harvey's Legal Agent” · Kimi K3: “Harvey LAB-AA”
Reported by 4 of 8
Vals Finance Agent
Can the model do an equity analyst's research from SEC filings?
- Made by
- Vals AI, 2026
- Size
- Hundreds of questions, private test split
- Graded by
- Expert rubric, partial credit
- Headline score
- Accuracy
The model gets tools to search and read SEC filings and a question an analyst would be asked, across nine categories from financial modelling to precedent transactions. It has to find the numbers, do the arithmetic and cite where they came from.
Answers are marked against expert rubrics with partial credit, with one exception: "dealbreaker" errors, like using the wrong fiscal year, cap the score no matter how much else is right.
question · invented company
What was Northwind's adjusted operating margin for its Cloud segment in fiscal 2025? Cite your source.
answer 31.4% (FY2024 10-K, Note 17)
Before comparing scores
- The test split is private and the judge model isn't disclosed, so outsiders can't rerun it. Published sizes disagree (927 vs 537 questions).
Field Guide ai agent · tool use
Sources Vals AI: Finance Agent v2
As printed Gemini 4 Argon: “Vals Finance Agent v2” · Muse Wonder: “Vals Finance Agent V2” · Mistral Large 4: “Vals Finance Agent v2” · Kimi K3: “Finance Agent v2”
Reported by 4 of 8
GDPval-AA
Whose deliverable would a judge rather have: this model's or another's?
- Made by
- OpenAI; run by Artificial Analysis, 2025
- Size
- 220 tasks, 44 occupations
- Graded by
- Pairwise model judges
- Headline score
- Elo rating
OpenAI's GDPval collected real work products (documents, slides, spreadsheets) from 44 occupations and had experts judge models against human professionals. Artificial Analysis took the 220 public tasks and turned them into a league.
Two models do the same task. A panel of judge models looks at both deliverables and picks the better one. Thousands of these matches become an Elo rating, the same system chess uses: beating a strong opponent moves you up more than beating a weak one.
task
Make a one-page PDF stage plot for a five-piece touring band: where everyone stands, plus the input and output lists for the venue's sound crew.
model A · stage_plot.pdf (the model being rated)
model B · stage_plot.pdf
model A rating
1724 +12
model B rating
1686 −12
Before comparing scores
- Elo is relative. A score only means something next to the other models rated in the same pool, so v2 and v2.1 numbers don't compare.
- The judges are models, and the 220 tasks are public, so they may have leaked into training data.
- SpaceXAI's chart just says "GDPval"; it's an Elo, so it is almost certainly this.
Field Guide benchmark · data leakage
Sources GDPval paper (arXiv 2510.04374) · Artificial Analysis: GDPval-AA
As printed Opus 5.5: “GDPval-AA v2.1” · Grok 4.7: “GDPval (Elo chart)” · GLM-5.3: “GDPval-AA v2” · Kimi K3: “GDPval-AA v2”
Reported by 3 of 8
HealthBench Professional
Are the model's answers to clinicians useful and safe?
- Made by
- OpenAI, 2026
- Size
- 525 tasks
- Graded by
- Physician rubric, ±10 points
- Headline score
- % of rubric points
190 physicians chose 525 real clinician conversations. For each one they wrote a rubric where every line carries points from −10 to +10: credit for what a good answer must say, penalties for what it must not (a wrong dose, an invented reference).
A grader model marks the answer against each line. The score is the points earned out of the points available, clipped at zero, and answers over 2,000 characters lose points, so padding doesn't pay.
clinician · invented example
Summarise these discharge notes for the patient's family doctor.
Discharged day 4 after pneumonia. Medication changes: amoxicillin course to finish on the 14th; ramipril paused; started omeprazole. Biopsy result pending; please follow up in 2 weeks…
1,640 characters · under the 2,000 limit, no length penalty
Before comparing scores
- The grader is OpenAI's own model (GPT-5.4), and the hard cases are oversampled.
- Some models decline some medical items, which counts against them.
- Meta labels its row "HealthBench Pro"; that's very likely this benchmark, but Meta doesn't say.
Field Guide benchmark · calibration
Sources HealthBench Professional paper (arXiv 2604.27470)
As printed GPT-6 Astra: “HealthBench Professional” · Muse Wonder: “HealthBench Pro” · Grok 4.7: “HealthBench Professional”
Reported by 0 of 8 Too new for launch tables
GDPx
Can the model do long, agentic work in the domains where frontier AI will earn the most?
- Made by
- SemiAnalysis + 14 data partners, 2026
- Size
- 206 tasks, 11 domains
- Graded by
- Per-domain checks, 3 attempts
- Headline score
- % (equal-weight domain average)
SemiAnalysis built GDPx as a successor to GDPval for agents. Instead of weighting jobs by share of GDP or the O*NET occupation list, it picks the 11 domains it expects to be accelerated most by AI in the next year or two: recursive self-improvement, chip design, hardware design, cyber, biology, finance, legal, quant trading, materials science, alignment and software engineering. Government, real estate and insurance are left out on purpose.
Each domain's tasks come from a specialist RL-environment company (Datacurve for software, Arithmetic for cyber, Latch Bio for biology, Fleet for alignment, and so on), and every task ships with a working environment, not just a prompt. One finance task drops the model into a deal room of about 2,000 anonymized files from a real transaction; cyber tasks are built on real zero-days with live services running. SemiAnalysis ran its own automated QA, and a partner had to pass it on the first try to be included.
Models run in their native harness where one exists (Claude Code, Codex, Antigravity; OpenCode otherwise) with a default limit of 8 hours. Domains grade differently, some pass/fail and some partial credit. The headline is pass@1 averaged over three attempts, with every domain weighted equally. At launch the best model scored 12%.
task · as Marbury Kline, the accounting advisor
Bannock Capital is buying about 62% of Haverly Snack Company for $880M plus an earnout. Do the accounting work the deal team asked for, using the data room and your firm's deal folder.
wrote deliverables/earnout_accounting.xlsx
Its own leaderboard · November 2, 2026
GDPx v1.0 headline score, all 11 domains, as published by SemiAnalysis.
- Claude Opus 5.5: 12.0%
- Gemini 4 Carbon: 10.9%
- GPT-6 Astra: 9.6%
- Muse Wonder Full: 9.0%
- Claude Fable 5.1: 8.5%
- GPT 6.1 Sol: 8.4%
- Claude Sonnet 5.5: 8.2%
- Gemini 4 Argon: 7.3%
- Muse Wonder: 6.8%
- DeepSeek 4.1 Flash 1017: 5.4%
- GPT-6 Luna: 5.1%
- Grok 4.7: 3.9%
- GLM 5.3: 2.7%
Before comparing scores
- Safety classifiers count as failures. SemiAnalysis says Opus 5.5 and GPT-6 Astra are pulled down by refusals on the cyber, biology and alignment tasks, and publishes a second ranking with those three domains removed. Check which ranking a chart shows.
- Equal domain weights make the headline very spiky. Finance and legal are so hard that almost every model scores 0 there, and Muse Wonder Full, fourth overall, is SemiAnalysis's pick for the best cyber model.
- It is new (v1.0, November 2026) and run by one analyst firm, not the labs. None of the eight launch tables on this page prints it yet.
- With 206 tasks across 11 domains, each domain is under 20 tasks on average, so a single domain score moves a lot with one task.
Field Guide benchmark · ai agent · ai safety
Sources SemiAnalysis: GDPx, measuring frontier model capabilities · GDPx leaderboard · Launch thread on X
Hard questions · 4 benchmarks
One question, one right answer
The oldest format: a fixed question with a known answer. Cheap to grade and easy to compare, which is exactly why most of them are now saturated.
Reported by 4 of 8
Humanity's Last Exam
Can the model answer questions at the edge of expert knowledge, and know when it can't?
- Made by
- Center for AI Safety & Scale AI, 2025
- Size
- 2,500 questions
- Graded by
- Exact answer, model judge
- Headline score
- Accuracy (+ calibration)
Nearly a thousand experts wrote questions in over a hundred subjects that the best models of early 2025 got wrong. Every question has one short, checkable answer; about a quarter are multiple choice and about 14% need an image.
Alongside the answer, the model states how confident it is. Accuracy is the headline, but HLE also reports calibration error: a model that says "95% sure" should be right about 95% of the time.
Launch tables now report HLE "with tools", where the model can search the web and run code. The best score in this set is 67.7.
question · invented, in HLE's format
In how many ways can a 2 × 10 strip be tiled with 1 × 2 dominoes? Give a single integer.
model's answer
55
stated confidence
92%
reference answer
89
calibration · across many questions
Before comparing scores
- With search, an answer posted anywhere online can leak into a run.
- An outside audit estimated that about 30% of the chemistry and biology answers are wrong, so the ceiling is below 100. Corrected (HLE-Verified) and rolling versions exist.
Field Guide benchmark · calibration · data leakage
Sources Humanity's Last Exam · HLE paper (arXiv 2501.14249)
As printed Opus 5.5: “Humanity's Last Exam (with tools)” · GPT-6 Astra: “HLE (with tools)” · GLM-5.3: “HLE (with tools)” · Kimi K3: “HLE-Full (with tools)”
Reported by 2 of 8
GPQA Diamond
Can the model answer PhD-level science questions that web search can't solve?
- Made by
- Rein et al. (NYU), 2023
- Size
- 198 questions
- Graded by
- Multiple choice
- Headline score
- Accuracy
Biology, physics and chemistry questions written by PhDs, each with four options. They were tested on people before they were accepted: experts in the field scored 65%, and skilled non-experts with unlimited web access scored 34%. That gap is what "Google-proof" means.
Diamond is the hardest 198 of 448 questions. Grading is the simplest on this page: did the model pick the right letter?
question · invented, and far easier than a real one
A photon has a wavelength of 500 nm. Its energy is closest to:
E = hc / λ ≈ 1240 eV·nm ÷ 500 nm = 2.48 eV → B
Before comparing scores
- It is saturated. The two launches that still report it score 93.5 and 96.0, and some questions are believed to be wrong, so the real ceiling is below 100. Six of the eight launches no longer print it.
- With 198 questions, one question is half a point. Differences under about three points are noise.
Field Guide benchmark · test set
Sources GPQA paper (arXiv 2311.12022)
As printed GPT-6 Astra: “GPQA Diamond” · Kimi K3: “GPQA Diamond”
Reported by 2 of 8
BrowseComp
Can the model keep searching the web until it finds a fact that is buried on purpose?
- Made by
- OpenAI, 2025
- Size
- 1,266 questions
- Graded by
- Short answer, model judge
- Headline score
- Accuracy
The questions were written backwards: start from an obscure fact, then describe it with a pile of constraints so that finding it takes dozens of searches. The answer is short and easy to check once found. Human trainers solved 29.2%.
What it measures is persistence and bookkeeping: trying a new angle after a dead end, and keeping track of which constraints a candidate meets.
question
Please identify the fictional character who occasionally breaks the fourth wall with the audience, has a backstory involving help from selfless ascetics, is known for his humor, and had a TV show that aired between the 1960s and 1980s with fewer than 50 episodes.
searches 23 · pages read 61
candidate: Plastic Man
Before comparing scores
- It is close to saturated (91.5 and 91.2 in the two launches that report it).
- The live web changes, so runs can't be reproduced exactly. BrowseComp-Plus fixes this with a frozen document collection.
Field Guide information retrieval · tool use · ai agent
Sources BrowseComp paper (arXiv 2504.12516)
As printed GPT-6 Astra: “BrowseComp” · Kimi K3: “BrowseComp”
Reported by 1 of 8
ARC-AGI
Can the model work out a new rule from a few examples, the way people do without trying?
- Made by
- ARC Prize Foundation, 2019
- Size
- 3 versions; private sets
- Graded by
- Exact grid match
- Headline score
- % solved
ARC-AGI-1 and -2 are grid puzzles. You see a few input→output pairs, work out the rule, and paint the output for a new input. Every cell has to be right. People find most of them easy; for years models didn't.
ARC-AGI-3 (March 2026) swapped static grids for small interactive games with no instructions: the model has to discover the goal by playing. At launch, frontier models scored 0.51% and people 100%.
example 1
example 2
test · the model paints the output
ARC-AGI-3 · GPT-6 Astra · two harnesses
Before comparing scores
- The harness is part of the result. OpenAI reports 99.9% on ARC-AGI-3 using its own adapter; the ARC Prize Foundation's standard harness scored the same model 62.7% on the semi-private set.
- ARC-AGI-1 and -2 are effectively solved (98.5 and 95.0 in OpenAI's table).
Field Guide benchmark · overfitting
Sources ARC Prize · ARC Prize: verifying GPT-6 Astra
As printed GPT-6 Astra: “ARC-AGI-1, -2 and -3”
Retired · 6
The numbers you won't see any more
A benchmark stops being useful once every frontier model aces it, or once its answers leak into training data. These used to headline launch posts. None of the 8 launches above prints them in its headline table, except where noted.
-
SWE-bench Verified
The coding number, 2024–25
OpenAI stopped reporting it in February 2026: most of the problems still unsolved had faulty tests, and models could reproduce the reference patch from the task ID alone. DeepSWE took its row. Latent Space: SWE-bench is dead
-
OSWorld-Verified
The computer-use number
Pushed past 80%. Replaced by OSWorld 2: Kimi K3 scores 84.8 on the old test and 58.3 on the new one.
-
GPQA Diamond
The science number
Still printed by two labs, at 93–96%. Kept on this page because you will still see it.
-
MMLU · MMLU-Pro
The general-knowledge number
Multiple choice across 57 subjects. None of the eight launches reports it any more.
-
AIME · HMMT
The maths number
Competition maths with integer answers. Absent from all eight launch tables; OpenAI now cites FrontierMath, which it says its model also saturates.
-
LiveCodeBench · MMMU · τ²-bench
Coding puzzles · visual questions · tool use
None of the eight launches reports them. Kimi K3 still reports the harder MMMU-Pro and τ³-bench's banking track.
Next time you read a launch post
Six questions before you trust a number
- Which version?
- Terminal-Bench 2.1, 3.0 and 4.0 share a name and nothing else. Older versions are easier. See it
- Which harness?
- The loop around the model is part of the result. One model scored 99.9% and 62.7% on the same ARC-AGI-3 tasks. See it
- Which score?
- Partial or binary, per-criterion or all-pass. The same run can be 81% or 0%. See it
- Who ran it?
- The lab, the benchmark's authors, or a third party ("-AA" is Artificial Analysis). Labs often print other labs' scores, and they rarely match. See it
- Who graded it?
- Tests and exact matches are objective. Rubrics and pairwise votes are marked by a model, sometimes the publishing lab's own. See it
- How many items?
- On a 66-task benchmark one task is 1.5 points. A two-point lead can be one lucky run. See it
When the tables change, you'll hear about it
New tests show up with every launch. Get a short note when one is worth understanding.
Check your inbox to confirm.