Benchmarks · 14 tests · 8 launches

Benchmarks

Every model launch ends in a table of scores. This is what's behind each row: how the test works, how one item gets graded, and why two numbers with the same name often don't compare.

Reported in each lab's latest launch
Benchmark Opus 5.5 GPT-6 Astra Gemini 4 Argon Muse Wonder Grok 4.7 Mistral Large 4 GLM-5.3 Kimi K3 of 8
Terminal-Bench reported as Terminal-Bench 4.0 reported as Terminal-Bench 4.0 reported as Terminal-Bench 4.0 reported as Terminal-Bench 4.0-AA reported as Terminal-Bench 4.0 reported as Terminal-Bench 4.0 reported as Terminal Bench 3.0 reported as Terminal-Bench 2.1 8
DeepSWE not reported reported as DeepSWE v1.1 reported as DeepSWE v1.1 reported as DeepSWE 1.1 reported as DeepSWE v1.1 reported as DeepSWE v1.1 reported as DeepSWE reported as DeepSWE 7
AutomationBench reported as AutomationBench (run by Zapier) reported as AutomationBench reported as AutomationBench (private set) reported as AutomationBench-AA not reported reported as AutomationBench (AA, 657 tasks) reported as AutomationBench reported as AutomationBench (600-task public set) 7
OSWorld 2 reported as OSWorld 2.1 (partial) reported as OSWorld 2.0 (offline, partial) reported as OSWorld-2.0 (offline, partial) reported as OSWorld V2.1 (partial and binary) not reported not reported not reported reported as OSWorld 2.0 5
Harvey LAB not reported not reported reported as Harvey's Legal Agent Benchmark reported as Harvey LAB-AA reported as Harvey Legal Agent Benchmark reported as Harvey's Legal Agent not reported reported as Harvey LAB-AA 5
ALE not reported reported as Agents' Last Exam reported as Agent's Last Exam not reported not reported not reported reported as Agents' Last Exam (CLI) reported as Agents' Last Exam 4
Vals Finance not reported not reported reported as Vals Finance Agent v2 reported as Vals Finance Agent V2 not reported reported as Vals Finance Agent v2 not reported reported as Finance Agent v2 4
GDPval-AA reported as GDPval-AA v2.1 not reported not reported not reported reported as GDPval (Elo chart) not reported reported as GDPval-AA v2 reported as GDPval-AA v2 4
Humanity's Last Exam reported as Humanity's Last Exam (with tools) reported as HLE (with tools) not reported not reported not reported not reported reported as HLE (with tools) reported as HLE-Full (with tools) 4
HealthBench Pro not reported reported as HealthBench Professional not reported reported as HealthBench Pro reported as HealthBench Professional not reported not reported not reported 3
GPQA Diamond not reported reported as GPQA Diamond not reported not reported not reported not reported not reported reported as GPQA Diamond 2
BrowseComp not reported reported as BrowseComp not reported not reported not reported not reported not reported reported as BrowseComp 2
ARC-AGI not reported reported as ARC-AGI-1, -2 and -3 not reported not reported not reported not reported not reported not reported 1
GDPx not reported not reported not reported not reported not reported not reported not reported not reported 0

in the launch post's headline table not reported

Checked October 15, 2026. One flagship per lab: Anthropic Claude Opus 5.5, OpenAI GPT-6 Astra, Google Gemini 4 Argon, Meta Muse Wonder, SpaceXAI Grok 4.7, Mistral Large 4, Z.ai GLM-5.3, Moonshot AI Kimi K3.

Before the tests

How a benchmark turns into a number

A benchmark is a fixed set of tasks and a rule for marking them. Run a model on every task, mark each one, and average: that's the number in the table. But the model never takes the test alone. A harness decides what tools it gets and how long it can try, and a grader decides what counts as right. Both are choices, and both move the score.

Each figure below shows one item going through a real benchmark, from the task to the grader's verdict. In every figure, orange is what the model wrote, and the grader's marks are green or red.

every benchmark five parts, in order assembling one number
  1. 1

    Dataset

    A frozen set of tasks with answers or tests kept out of sight.

    66 tasks, tests hidden

  2. 2

    Harness

    The loop around the model: its tools, time limit, retries and prompt.

    a shell, 8 hours, 5 runs

  3. 3

    Model

    The only part a launch table names.

    Opus · GPT · Gemini · Grok…

  4. 4

    Grader

    Decides each item: tests, exact match, a rubric, or another model.

    hidden tests

  5. 5

    Score

    One number per benchmark, averaged over items (and often runs).

    % of tasks resolved

change any box except the model and the number changes too
The small print in each box follows Terminal-Bench. Swap the harness or the grader and the score moves, even with the same model.

Coding · 2 benchmarks

Write the code, pass the hidden tests

The model gets a real environment and a task in plain words. It works until it says it is done; then tests it never saw decide.

Reported by 8 of 8

Terminal-Bench

Can the model finish hard, hours-long jobs in a real Linux terminal?

Made by
Stanford & Laude Institute, 2025
Size
66 tasks (v4.0)
Graded by
Hidden tests
Headline score
% of tasks resolved

Each task is a Docker container, an instruction file and a set of tests the model never sees. The model gets a shell and works: installs packages, writes code, runs it, reads errors. It can take hours; the limit is up to eight. When it says it is done, the hidden tests run against whatever state it left behind.

A task counts only if every test passes. The headline number is the share of tasks resolved, and the public leaderboard runs each task five times because a single run is noisy.

Version 4.0 (August 2026) has 66 tasks. Eight of version 3.0's 74 were removed: two were too easy, two drew refusals, two had solutions posted online, and two had quality problems.

terminal-bench 4.0 task vf2-speedup-networkx · 1 of 66 agent working hidden tests failed

instruction.md

Build fast_networkx, a faster drop-in subset of NetworkX focused on VF2++ graph isomorphism. import fast_networkx as fnx must work from /app.

$ ls /app
README.md bench.py
$ python -m cProfile -s cumtime bench.py | head -4
91% of time in isomorphvf2pp.py: _feasibility()
$ mkdir fast_networkx && cat > fast_networkx/_core.pyx <<'EOF'
… 412 lines
$ cythonize -i fast_networkx/_core.pyx
built fast_networkx/_core.so
$ python bench.py
fast_networkx vs networkx: 1,830x faster

agent stopped · hidden tests run

test_import_from_app PASSED
test_matches_networkx_on_10k_graphs PASSED
test_speedup_at_least_5000x FAILED · measured 1,830x
✗ task failed 2 of 3 tests passed. The task still scores 0.
score = tasks where every hidden test passes ÷ 66
One Terminal-Bench 4.0 task, vf2-speedup-networkx: the instruction is quoted from the task; the agent's commands and the measured speedup are illustrative.

Before comparing scores

  • Versions don't compare. All eight launches print "Terminal-Bench", but Kimi K3 reports 2.1 and GLM-5.3 reports 3.0, older and easier versions than everyone else's 4.0.
  • The agent harness around the model (Terminus, Claude Code, Codex…) and its effort setting move the score by several points. The same model appears with three different 4.0 scores in three labs' tables.
  • Meta's "-AA" row is a separate run by Artificial Analysis, not Meta's own.
  • With 66 tasks, one task is about 1.5 points. Differences of two or three points are inside the noise.

Field Guide benchmark · ai agent · tool use
Sources Terminal-Bench 4.0 announcement · Terminal-Bench repository (Harbor) · Terminal-Bench paper (v2.0, arXiv 2601.11868)
As printed Opus 5.5: “Terminal-Bench 4.0” · GPT-6 Astra: “Terminal-Bench 4.0” · Gemini 4 Argon: “Terminal-Bench 4.0” · Muse Wonder: “Terminal-Bench 4.0-AA” · Grok 4.7: “Terminal-Bench 4.0” · Mistral Large 4: “Terminal-Bench 4.0” · GLM-5.3: “Terminal Bench 3.0” · Kimi K3: “Terminal-Bench 2.1”

Reported by 7 of 8

DeepSWE

Can the model ship a real feature or fix in a real codebase, the way a teammate would?

Made by
Datacurve, 2026
Size
113 tasks, 91 repos
Graded by
Hidden + regression tests
Headline score
pass@1

DeepSWE is the coding benchmark that replaced SWE-bench in launch tables. Each task is a short, human-sounding request ("fix the label sorting") and a real open-source repository in TypeScript, Go, Python, JavaScript or Rust. The model gets bash and nothing else, and writes a patch.

Two test suites grade it. New tests, written for the task and hidden from the model, check the behaviour through the project's public API. Then the repository's own test suite runs, so a fix that breaks something else fails.

The tasks were written from scratch for the benchmark, so the answer can't already be in a model's training data. They are big: the reference solutions average about 668 lines across seven files. The score is pass@1: solved on the first attempt.

deepswe v1.1 1 of 113 · real repo, task written from scratch writing patch running tests solved

request

Fix PromQL label sorting across typed and untyped values

promql/functions.go
− sort.Slice(out, func(i, j int) bool {
− return out[i].Value < out[j].Value
+ sort.SliceStable(out, func(i, j int) bool {
+ return labels.Compare(out[i], out[j]) < 0
})
model/labels/compare.go
+ // Compare orders typed values numerically and
+ // untyped values lexically, typed first.
+ func Compare(a, b Label) int {
+ if a.Typed() != b.Typed() { return boolCmp(a.Typed()) }

new tests · hidden from the model

12 / 12 passed

checks the new behaviour through the public API

the repo's own test suite

1,284 / 1,284 passed

proves the fix didn't break anything else

✓ solved on the first try This attempt counts toward pass@1.
pass@1 = tasks solved on the first attempt ÷ 113
A DeepSWE v1.1 task: the title is quoted from the public task list; the patch and test names are illustrative.

Before comparing scores

  • Labs mix their own runs with numbers from Datacurve's leaderboard, and the top models sit in a narrow 74–79 band where harness choices matter more than the gap.
  • Version 1.1 (September 2026) changed tasks from v1. Check which one a table means.

Field Guide benchmark · test set · data leakage
Sources DeepSWE (Datacurve)
As printed GPT-6 Astra: “DeepSWE v1.1” · Gemini 4 Argon: “DeepSWE v1.1” · Muse Wonder: “DeepSWE 1.1” · Grok 4.7: “DeepSWE v1.1” · Mistral Large 4: “DeepSWE v1.1” · GLM-5.3: “DeepSWE” · Kimi K3: “DeepSWE”

Computer work · 3 benchmarks

Use the software, change the state

No code to grade here. The model clicks, calls APIs and edits files, and the grader inspects what the world looks like afterwards.

Reported by 7 of 8

AutomationBench

Can the model run a business workflow across many apps without breaking the rules?

Made by
Zapier, 2026
Size
600 public tasks + private set
Graded by
Final-state check
Headline score
% of tasks passed

The model works inside simulated versions of everyday SaaS apps (calendars, spreadsheets, Slack, CRMs: 47 tools and about 500 API endpoints) and is given a workflow to finish. The data contains decoys: an outdated policy sheet, a similarly named channel.

Nobody reads the model's reasoning. When it stops, a program inspects the state of every app. Each task has must-pass objectives (the right event was renamed, the summary was posted) and must-not-happen guardrails (nothing deleted, nothing sent to the wrong place). Miss an objective or trip a guardrail and the task fails.

automationbench 1 of 600 public · 47 simulated apps agent calling APIs checking final state passed

request

There's a scheduling conflict on February 20, 2026 at 2:00 PM — a Zoom meeting and a Google Calendar event overlap. Check the meeting priority policy in the spreadsheet to determine which one wins, then reschedule the loser by prepending [RESCHEDULED] to its topic/title. Post a summary to #ops-updates on Slack…

what the model did

GET /zoom/meetings?date=2026-02-20
GET /gcal/events?date=2026-02-20
GET /sheets/ops/Meeting priority policy (2024, archived) decoy: archived copy, ignored
GET /sheets/ops/Meeting priority policy
PATCH /gcal/events/evt_31 title: "[RESCHEDULED] Vendor sync"
POST /slack/chat.postMessage #ops-updates

what the grader checks after

✓ objective · losing event renamed with [RESCHEDULED]
✓ objective · winning meeting left untouched
✓ objective · summary posted in #ops-updates
✓ guardrail · nothing deleted
✓ guardrail · no post to any other channel
✓ task passed Nobody read the model's reasoning. Only the apps' final state counted.
pass = every objective met and no guardrail tripped
An AutomationBench task: the request is quoted from the paper; the API calls and checks are illustrative.

Before comparing scores

  • Three different things share the name: the 600-task public set, Zapier's private set, and Artificial Analysis's 657-task "-AA" version, which scores the share of objectives met and zeroes any task with a guardrail violation.
  • Anthropic's numbers come from Zapier's own runs, where a safety refusal counts as a failure.

Field Guide ai agent · tool use
Sources AutomationBench paper (arXiv 2604.18934) · Zapier benchmarks
As printed Opus 5.5: “AutomationBench (run by Zapier)” · GPT-6 Astra: “AutomationBench” · Gemini 4 Argon: “AutomationBench (private set)” · Muse Wonder: “AutomationBench-AA” · Mistral Large 4: “AutomationBench (AA, 657 tasks)” · GLM-5.3: “AutomationBench” · Kimi K3: “AutomationBench (600-task public set)”

Reported by 5 of 8

OSWorld 2

Can the model use a real desktop, across many apps, for an hour and a half of work?

Made by
XLANG Lab (HKU), 2026
Size
108 tasks
Graded by
~27 checkpoints per task
Headline score
Partial or binary %

The model sees screenshots of a virtual machine and controls it with mouse and keyboard. Tasks take a person a median of about 1.6 hours and an agent around 300 actions. They span apps (a PDF, a web portal, email, a bank site), and some change halfway: a new email arrives with information the model has to act on.

Each task has about 27 checkpoints. The "partial" score is the average share of checkpoints passed; the "binary" score only counts tasks where every checkpoint passed.

OSWorld 2 replaced the original OSWorld-Verified, which top models had pushed past 80%.

osworld 2.1 1 of 108 · a human needs ~1.6 hours agent at the desktop scored

task

File last month's expense claim in the company portal. Follow the steps in the tutorial PDF, read the amounts off the receipts, and check them against Gmail and the bank statement.

tutorial.pdf expense portal gmail bank

actions taken 312

new email · mid-task "Use cost centre 4410, not 4400."

checkpoints

partial score

81%

22 of 27 checkpoints. Most launch tables print this one.

binary score

0%

Five checkpoints missed, so the task wasn't done.

partial = mean share of checkpoints passed · binary = share of tasks with every checkpoint passed
An OSWorld 2 task, paraphrased from the paper's example (an expense claim with a mid-task email); the checkpoint results are illustrative.

Before comparing scores

  • Partial and binary are very different numbers. Meta reports both for Muse Wonder Full: 78.4 partial, 39.4 binary. Most labs print only partial.
  • Labs pick online or offline subsets and different releases (2.0, the August patch that fixed reward hacking in 14 tasks, 2.1).
  • Kimi K3 scores 84.8 on the old OSWorld-Verified and 58.3 on OSWorld 2.0. Same model, newer test.

Field Guide ai agent · computer vision
Sources OSWorld 2 repository and paper (arXiv 2606.29537)
As printed Opus 5.5: “OSWorld 2.1 (partial)” · GPT-6 Astra: “OSWorld 2.0 (offline, partial)” · Gemini 4 Argon: “OSWorld-2.0 (offline, partial)” · Muse Wonder: “OSWorld V2.1 (partial and binary)” · Kimi K3: “OSWorld 2.0”

Reported by 4 of 8

Agents' Last Exam

Can the model do a professional's computer work in the software professionals use?

Made by
Berkeley RDI + 250 experts, 2026
Size
1,490 tasks (152 public)
Graded by
Compare to expert output
Headline score
Full-pass %

Experts from 13 industries wrote tasks in the real tools of their trade, from video compositing to CAD. The model produces files, and each file is compared against an expert's reference output or rubric, field by field, with tolerances where numbers are involved.

A task passes only if the whole output passes. Tasks come in three tiers; on the hardest one, models average under 1%. There is also a terminal-only subset, ALE-CLI.

agents' last exam public set · 1 of 152 agent extracting comparing to reference failed

task

Pull the headline figures out of this 10-K filing into extraction.json.

filing_10k.pdf · 212 pages → extraction.json
field model expert check
revenue_fy2025 48.21B 48.21B ✓ exact
operating_income 9.87B 9.84B ✓ within ±0.5%
diluted_eps 3.12 3.12 ✓ exact
segments 4 4 ✓ exact
long_term_debt 12.4B 14.1B ✗ missed a note
✗ not a full pass 4 of 5 fields right. A professional's file with one wrong number is a wrong file.
score = tasks whose whole output matches the expert's ÷ tasks attempted
An ALE-style extraction task, paraphrased from the public set (figures pulled from an SEC 10-K); the field values are illustrative.

Before comparing scores

  • The pool keeps growing and has tiers and a CLI subset, and launch posts rarely say which slice they ran. GLM-5.3 labels its row "(CLI)"; the four labs' numbers range from 28 to 59 and almost certainly aren't measuring the same thing.

Field Guide ai agent · tool use
Sources Agents' Last Exam · ALE paper (arXiv 2606.05405)
As printed GPT-6 Astra: “Agents' Last Exam” · Gemini 4 Argon: “Agent's Last Exam” · GLM-5.3: “Agents' Last Exam (CLI)” · Kimi K3: “Agents' Last Exam”

Professional work · 5 benchmarks

Produce the deliverable, face the rubric

A memo, a model, a clinical answer. There is no single right string, so experts write checklists, and a judge (often another model) marks against them.

Reported by 5 of 8

Harvey's Legal Agent Benchmark

Can the model produce legal work a partner would accept for review?

Made by
Harvey, 2026
Size
1,200+ tasks, 75,000+ criteria
Graded by
Rubric, model judge
Headline score
All-criteria pass %

Each task reads like a partner's instruction to an associate, plus a closed set of client documents: a data room, contracts, correspondence. The model writes the memo or draft.

Lawyers wrote a rubric for every task, more than 75,000 criteria in all. A judge model checks the output against each one. The strict score counts a task only if every criterion passes, which is why all-pass scores sit between roughly 4% and 20%.

harvey lab 1 of 1,200+ · 24 practice areas agent drafting judge marking rubric failed

partner's instruction + data room (37 documents)

Review the data room, identify change-of-control provisions, assess the risk to the deal, recommend next steps, and prepare a review-ready draft memorandum for the deal team and Board.

draft memorandum

DRAFT · Privileged. Section 14.2 of the Supply Agreement treats the merger as a change of control requiring the supplier's written consent on 30 days' notice; absent consent the supplier may terminate. We recommend…

✓ finds the change-of-control clause in the supply agreement
✓ notes the consent requirement and its 30-day notice
✓ flags the counterparty termination right as a deal risk
✓ cites each provision by section
✗ catches the second trigger in the credit facility
✓ recommends seeking consent before signing
✓ addressed to the deal team and the Board
✓ marked as a draft for review

per criterion

88%

7 of 8 criteria met. Kimi K3's 94.6 is this kind of number.

all-pass (LAB's score)

0

One criterion missed fails the task. Top all-pass scores are roughly 4–20%.

LAB score = tasks where every rubric criterion passes ÷ tasks
A LAB task: the instruction is quoted from Harvey's announcement; the memo and the rubric lines are illustrative.

Before comparing scores

  • Kimi K3 reports 94.6, which is the share of individual criteria met, not tasks fully passed. The two numbers can't be compared.
  • LAB-AA is Artificial Analysis's private 120-task version with a single judge model; Vals runs another. The score depends on the judge.

Field Guide benchmark · ai agent
Sources Introducing Harvey's Legal Agent Benchmark · harvey-labs repository
As printed Gemini 4 Argon: “Harvey's Legal Agent Benchmark” · Muse Wonder: “Harvey LAB-AA” · Grok 4.7: “Harvey Legal Agent Benchmark” · Mistral Large 4: “Harvey's Legal Agent” · Kimi K3: “Harvey LAB-AA”

Reported by 4 of 8

Vals Finance Agent

Can the model do an equity analyst's research from SEC filings?

Made by
Vals AI, 2026
Size
Hundreds of questions, private test split
Graded by
Expert rubric, partial credit
Headline score
Accuracy

The model gets tools to search and read SEC filings and a question an analyst would be asked, across nine categories from financial modelling to precedent transactions. It has to find the numbers, do the arithmetic and cite where they came from.

Answers are marked against expert rubrics with partial credit, with one exception: "dealbreaker" errors, like using the wrong fiscal year, cap the score no matter how much else is right.

vals finance agent v2 financial modelling · test split is private agent researching grading scored 0

question · invented company

What was Northwind's adjusted operating margin for its Cloud segment in fiscal 2025? Cite your source.

→ search_filings("NORTHWIND 10-K")
→ read("10-K FY2024", "Note 17 · Segments")
→ calc((4.20 + 0.31) / 14.37)

answer 31.4% (FY2024 10-K, Note 17)

✓ uses Cloud segment revenue, not total revenue 1 / 1
✓ adds back the one-off restructuring charge 1 / 1
✓ cites the filing and note 1 / 1
✗ states how the margin is defined 0 / 1
! dealbreaker · answered from the FY2024 filing; the question asked about FY2025 cap
0.75 0 3 of 4 points would be 0.75 partial credit. The wrong fiscal year caps the whole answer at zero.
accuracy = mean rubric credit across questions, with dealbreakers scoring 0
A finance-agent question, paraphrased (the real test split is private); the filings, numbers and rubric are invented.

Before comparing scores

  • The test split is private and the judge model isn't disclosed, so outsiders can't rerun it. Published sizes disagree (927 vs 537 questions).

Field Guide ai agent · tool use
Sources Vals AI: Finance Agent v2
As printed Gemini 4 Argon: “Vals Finance Agent v2” · Muse Wonder: “Vals Finance Agent V2” · Mistral Large 4: “Vals Finance Agent v2” · Kimi K3: “Finance Agent v2”

Reported by 4 of 8

GDPval-AA

Whose deliverable would a judge rather have: this model's or another's?

Made by
OpenAI; run by Artificial Analysis, 2025
Size
220 tasks, 44 occupations
Graded by
Pairwise model judges
Headline score
Elo rating

OpenAI's GDPval collected real work products (documents, slides, spreadsheets) from 44 occupations and had experts judge models against human professionals. Artificial Analysis took the 220 public tasks and turned them into a league.

Two models do the same task. A panel of judge models looks at both deliverables and picks the better one. Thousands of these matches become an Elo rating, the same system chess uses: beating a strong opponent moves you up more than beating a weak one.

gdpval-aa v2.1 1 of 220 · audio technician both models working judges comparing ratings updated

task

Make a one-page PDF stage plot for a five-piece touring band: where everyone stands, plus the input and output lists for the venue's sound crew.

model A · stage_plot.pdf (the model being rated)

model B · stage_plot.pdf

judge 1 → A judge 2 → A judge 3 → B A wins 2–1

model A rating

1724 +12

model B rating

1686 −12

Elo: thousands of matches like this one; beating a higher-rated model earns more
A GDPval-style task, paraphrased (a stage plot for a touring band); the votes and rating changes are illustrative.

Before comparing scores

  • Elo is relative. A score only means something next to the other models rated in the same pool, so v2 and v2.1 numbers don't compare.
  • The judges are models, and the 220 tasks are public, so they may have leaked into training data.
  • SpaceXAI's chart just says "GDPval"; it's an Elo, so it is almost certainly this.

Field Guide benchmark · data leakage
Sources GDPval paper (arXiv 2510.04374) · Artificial Analysis: GDPval-AA
As printed Opus 5.5: “GDPval-AA v2.1” · Grok 4.7: “GDPval (Elo chart)” · GLM-5.3: “GDPval-AA v2” · Kimi K3: “GDPval-AA v2”

Reported by 3 of 8

HealthBench Professional

Are the model's answers to clinicians useful and safe?

Made by
OpenAI, 2026
Size
525 tasks
Graded by
Physician rubric, ±10 points
Headline score
% of rubric points

190 physicians chose 525 real clinician conversations. For each one they wrote a rubric where every line carries points from −10 to +10: credit for what a good answer must say, penalties for what it must not (a wrong dose, an invented reference).

A grader model marks the answer against each line. The score is the points earned out of the points available, clipped at zero, and answers over 2,000 characters lose points, so padding doesn't pay.

healthbench professional 1 of 525 · chosen by physicians model answering grader marking scored

clinician · invented example

Summarise these discharge notes for the patient's family doctor.

Discharged day 4 after pneumonia. Medication changes: amoxicillin course to finish on the 14th; ramipril paused; started omeprazole. Biopsy result pending; please follow up in 2 weeks…

1,640 characters · under the 2,000 limit, no length penalty

● lists every medication change from the notes +7
● flags the biopsy result that is still pending +5
● states when follow-up is due +4
○ avoids unexplained abbreviations +3
● adds a medication that is not in the notes −6
○ invents a lab value or a reference −8
earned +7 +5 +4 −6 = 10 ÷ possible 19 = 53%
score = points earned (floored at 0) ÷ positive points available, averaged over items
A HealthBench-style item, invented for illustration (the real conversations come from clinicians); the rubric points show the scoring rule.

Before comparing scores

  • The grader is OpenAI's own model (GPT-5.4), and the hard cases are oversampled.
  • Some models decline some medical items, which counts against them.
  • Meta labels its row "HealthBench Pro"; that's very likely this benchmark, but Meta doesn't say.

Field Guide benchmark · calibration
Sources HealthBench Professional paper (arXiv 2604.27470)
As printed GPT-6 Astra: “HealthBench Professional” · Muse Wonder: “HealthBench Pro” · Grok 4.7: “HealthBench Professional”

Reported by 0 of 8 Too new for launch tables

GDPx

Can the model do long, agentic work in the domains where frontier AI will earn the most?

Made by
SemiAnalysis + 14 data partners, 2026
Size
206 tasks, 11 domains
Graded by
Per-domain checks, 3 attempts
Headline score
% (equal-weight domain average)

SemiAnalysis built GDPx as a successor to GDPval for agents. Instead of weighting jobs by share of GDP or the O*NET occupation list, it picks the 11 domains it expects to be accelerated most by AI in the next year or two: recursive self-improvement, chip design, hardware design, cyber, biology, finance, legal, quant trading, materials science, alignment and software engineering. Government, real estate and insurance are left out on purpose.

Each domain's tasks come from a specialist RL-environment company (Datacurve for software, Arithmetic for cyber, Latch Bio for biology, Fleet for alignment, and so on), and every task ships with a working environment, not just a prompt. One finance task drops the model into a deal room of about 2,000 anonymized files from a real transaction; cyber tasks are built on real zero-days with live services running. SemiAnalysis ran its own automated QA, and a partner had to pass it on the first try to be included.

Models run in their native harness where one exists (Claude Code, Codex, Antigravity; OpenCode otherwise) with a default limit of 8 hours. Domains grade differently, some pass/fail and some partial credit. The headline is pass@1 averaged over three attempts, with every domain weighted equally. At launch the best model scored 12%.

gdpx v1.0 finance · 1 of 206 tasks · 8 h limit agent working grading 3 attempts averaged

task · as Marbury Kline, the accounting advisor

Bannock Capital is buying about 62% of Haverly Snack Company for $880M plus an earnout. Do the accounting work the deal team asked for, using the data room and your firm's deal folder.

$ ls vdr/ # Haverly data room, ~2,000 files
$ grep -ril "earnout" vdr/legal/ marbury/
$ open("vdr/finance/QoE_draft_v3.xlsx")

wrote deliverables/earnout_accounting.xlsx

attempt 1 → fail attempt 2 → pass attempt 3 → fail pass@1 = 1 / 3
· finance this task + the rest
· legal
· cyber
· … 8 more
GDPx = mean of 11 domain scores, each weighted equally; each task averaged over 3 attempts
A GDPx finance task paraphrased from SemiAnalysis's launch post (an accounting request inside a private-equity deal room); the file reads, attempts and scores are illustrative.

Its own leaderboard · November 2, 2026

GDPx v1.0 headline score, all 11 domains, as published by SemiAnalysis.

  1. Claude Opus 5.5: 12.0%
  2. Gemini 4 Carbon: 10.9%
  3. GPT-6 Astra: 9.6%
  4. Muse Wonder Full: 9.0%
  5. Claude Fable 5.1: 8.5%
  6. GPT 6.1 Sol: 8.4%
  7. Claude Sonnet 5.5: 8.2%
  8. Gemini 4 Argon: 7.3%
  9. Muse Wonder: 6.8%
  10. DeepSeek 4.1 Flash 1017: 5.4%
  11. GPT-6 Luna: 5.1%
  12. Grok 4.7: 3.9%
  13. GLM 5.3: 2.7%

Full leaderboard

Before comparing scores

  • Safety classifiers count as failures. SemiAnalysis says Opus 5.5 and GPT-6 Astra are pulled down by refusals on the cyber, biology and alignment tasks, and publishes a second ranking with those three domains removed. Check which ranking a chart shows.
  • Equal domain weights make the headline very spiky. Finance and legal are so hard that almost every model scores 0 there, and Muse Wonder Full, fourth overall, is SemiAnalysis's pick for the best cyber model.
  • It is new (v1.0, November 2026) and run by one analyst firm, not the labs. None of the eight launch tables on this page prints it yet.
  • With 206 tasks across 11 domains, each domain is under 20 tasks on average, so a single domain score moves a lot with one task.

Field Guide benchmark · ai agent · ai safety
Sources SemiAnalysis: GDPx, measuring frontier model capabilities · GDPx leaderboard · Launch thread on X

Hard questions · 4 benchmarks

One question, one right answer

The oldest format: a fixed question with a known answer. Cheap to grade and easy to compare, which is exactly why most of them are now saturated.

Reported by 4 of 8

Humanity's Last Exam

Can the model answer questions at the edge of expert knowledge, and know when it can't?

Made by
Center for AI Safety & Scale AI, 2025
Size
2,500 questions
Graded by
Exact answer, model judge
Headline score
Accuracy (+ calibration)

Nearly a thousand experts wrote questions in over a hundred subjects that the best models of early 2025 got wrong. Every question has one short, checkable answer; about a quarter are multiple choice and about 14% need an image.

Alongside the answer, the model states how confident it is. Accuracy is the headline, but HLE also reports calibration error: a model that says "95% sure" should be right about 95% of the time.

Launch tables now report HLE "with tools", where the model can search the web and run code. The best score in this set is 67.7.

humanity's last exam 1 of 2,500 · mathematics model answering judge comparing wrong

question · invented, in HLE's format

In how many ways can a 2 × 10 strip be tiled with 1 × 2 dominoes? Give a single integer.

model's answer

55

stated confidence

92%

reference answer

89

✗ wrong An off-by-one: 55 is the answer for a 2 × 9 strip. Close doesn't count.

calibration · across many questions

said 50% sure right 41%
said 70% sure right 52%
said 90% sure right 61%
accuracy = answers matching the reference ÷ 2,500 · calibration error = gap between stated confidence and accuracy
An invented question in HLE's format; real HLE questions are much harder and kept out of public pages.

Before comparing scores

  • With search, an answer posted anywhere online can leak into a run.
  • An outside audit estimated that about 30% of the chemistry and biology answers are wrong, so the ceiling is below 100. Corrected (HLE-Verified) and rolling versions exist.

Field Guide benchmark · calibration · data leakage
Sources Humanity's Last Exam · HLE paper (arXiv 2501.14249)
As printed Opus 5.5: “Humanity's Last Exam (with tools)” · GPT-6 Astra: “HLE (with tools)” · GLM-5.3: “HLE (with tools)” · Kimi K3: “HLE-Full (with tools)”

Reported by 2 of 8

GPQA Diamond

Can the model answer PhD-level science questions that web search can't solve?

Made by
Rein et al. (NYU), 2023
Size
198 questions
Graded by
Multiple choice
Headline score
Accuracy

Biology, physics and chemistry questions written by PhDs, each with four options. They were tested on people before they were accepted: experts in the field scored 65%, and skilled non-experts with unlimited web access scored 34%. That gap is what "Google-proof" means.

Diamond is the hardest 198 of 448 questions. Grading is the simplest on this page: did the model pick the right letter?

gpqa diamond 1 of 198 · physics model answering correct

question · invented, and far easier than a real one

A photon has a wavelength of 500 nm. Its energy is closest to:

A 1.2 eV
B 2.5 eV
C 4.1 eV
D 12.4 eV

E = hc / λ ≈ 1240 eV·nm ÷ 500 nm = 2.48 eV → B

✓ B is correct Worth 1 of 198 questions: about half a point.
non-experts, with the web 34%
PhD experts in the field 65%
launch tables, 2026 93.5–96.0
accuracy = correct letters ÷ 198 · ±3 points is noise at this size
An invented question in GPQA's format, far easier than a real one. The authors ask that real items not be posted online.

Before comparing scores

  • It is saturated. The two launches that still report it score 93.5 and 96.0, and some questions are believed to be wrong, so the real ceiling is below 100. Six of the eight launches no longer print it.
  • With 198 questions, one question is half a point. Differences under about three points are noise.

Field Guide benchmark · test set
Sources GPQA paper (arXiv 2311.12022)
As printed GPT-6 Astra: “GPQA Diamond” · Kimi K3: “GPQA Diamond”

Reported by 2 of 8

BrowseComp

Can the model keep searching the web until it finds a fact that is buried on purpose?

Made by
OpenAI, 2025
Size
1,266 questions
Graded by
Short answer, model judge
Headline score
Accuracy

The questions were written backwards: start from an obscure fact, then describe it with a pile of constraints so that finding it takes dozens of searches. The answer is short and easy to check once found. Human trainers solved 29.2%.

What it measures is persistence and bookkeeping: trying a new angle after a dead end, and keeping track of which constraints a candidate meets.

browsecomp 1 of 1,266 · trainers solved 29% agent searching correct

question

Please identify the fictional character who occasionally breaks the fourth wall with the audience, has a backstory involving help from selfless ascetics, is known for his humor, and had a TV show that aired between the 1960s and 1980s with fewer than 50 episodes.

searches 23 · pages read 61

⌕ cartoon character breaks fourth wall raised by monks
✗ 0 candidates meet every clue
⌕ superhero backstory nursed back to health monastery
✗ 3 candidates
⌕ superhero comedy cartoon 1979 episode count
→ The Plastic Man Comedy/Adventure Show

candidate: Plastic Man

✓ breaks the fourth wall
✓ helped by selfless ascetics
✓ known for his humour
✓ TV show between the 1960s and 1980s
✓ fewer than 50 episodes
✓ matches: Plastic Man Hard to find, easy to check: that asymmetry is the whole design.
accuracy = answers a judge model matches to the reference ÷ 1,266
A BrowseComp question quoted from the paper; the search trail is illustrative.

Before comparing scores

  • It is close to saturated (91.5 and 91.2 in the two launches that report it).
  • The live web changes, so runs can't be reproduced exactly. BrowseComp-Plus fixes this with a frozen document collection.

Field Guide information retrieval · tool use · ai agent
Sources BrowseComp paper (arXiv 2504.12516)
As printed GPT-6 Astra: “BrowseComp” · Kimi K3: “BrowseComp”

Reported by 1 of 8

ARC-AGI

Can the model work out a new rule from a few examples, the way people do without trying?

Made by
ARC Prize Foundation, 2019
Size
3 versions; private sets
Graded by
Exact grid match
Headline score
% solved

ARC-AGI-1 and -2 are grid puzzles. You see a few input→output pairs, work out the rule, and paint the output for a new input. Every cell has to be right. People find most of them easy; for years models didn't.

ARC-AGI-3 (March 2026) swapped static grids for small interactive games with no instructions: the model has to discover the goal by playing. At launch, frontier models scored 0.51% and people 100%.

arc-agi invented puzzle · real sets are private model painting exact match

example 1

→

example 2

→

test · the model paints the output

→
✓ 16 of 16 cells One wrong cell would have made it a miss. There is no partial credit.

ARC-AGI-3 · GPT-6 Astra · two harnesses

OpenAI's own adapter 99.9%
ARC Prize standard harness 62.7%
score (ARC-AGI-1 and -2) = tasks whose output matches every cell ÷ tasks
An invented ARC-style grid puzzle (the real evaluation sets are private).

Before comparing scores

  • The harness is part of the result. OpenAI reports 99.9% on ARC-AGI-3 using its own adapter; the ARC Prize Foundation's standard harness scored the same model 62.7% on the semi-private set.
  • ARC-AGI-1 and -2 are effectively solved (98.5 and 95.0 in OpenAI's table).

Field Guide benchmark · overfitting
Sources ARC Prize · ARC Prize: verifying GPT-6 Astra
As printed GPT-6 Astra: “ARC-AGI-1, -2 and -3”

Retired · 6

The numbers you won't see any more

A benchmark stops being useful once every frontier model aces it, or once its answers leak into training data. These used to headline launch posts. None of the 8 launches above prints them in its headline table, except where noted.

  • SWE-bench Verified

    The coding number, 2024–25

    OpenAI stopped reporting it in February 2026: most of the problems still unsolved had faulty tests, and models could reproduce the reference patch from the task ID alone. DeepSWE took its row. Latent Space: SWE-bench is dead

  • OSWorld-Verified

    The computer-use number

    Pushed past 80%. Replaced by OSWorld 2: Kimi K3 scores 84.8 on the old test and 58.3 on the new one.

  • GPQA Diamond

    The science number

    Still printed by two labs, at 93–96%. Kept on this page because you will still see it.

  • MMLU · MMLU-Pro

    The general-knowledge number

    Multiple choice across 57 subjects. None of the eight launches reports it any more.

  • AIME · HMMT

    The maths number

    Competition maths with integer answers. Absent from all eight launch tables; OpenAI now cites FrontierMath, which it says its model also saturates.

  • LiveCodeBench · MMMU · τ²-bench

    Coding puzzles · visual questions · tool use

    None of the eight launches reports them. Kimi K3 still reports the harder MMMU-Pro and τ³-bench's banking track.

Next time you read a launch post

Six questions before you trust a number

Which version?
Terminal-Bench 2.1, 3.0 and 4.0 share a name and nothing else. Older versions are easier. See it
Which harness?
The loop around the model is part of the result. One model scored 99.9% and 62.7% on the same ARC-AGI-3 tasks. See it
Which score?
Partial or binary, per-criterion or all-pass. The same run can be 81% or 0%. See it
Who ran it?
The lab, the benchmark's authors, or a third party ("-AA" is Artificial Analysis). Labs often print other labs' scores, and they rarely match. See it
Who graded it?
Tests and exact matches are objective. Rubrics and pairwise votes are marked by a model, sometimes the publishing lab's own. See it
How many items?
On a 66-task benchmark one task is 1.5 points. A two-point lead can be one lucky run. See it

When the tables change, you'll hear about it

New tests show up with every launch. Get a short note when one is worth understanding.