Evaluation design. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.
For: Developers and teams putting AI into practical use
⚡ TL;DR — the 30-second version
- Compare the exact benchmark version and run conditions.
- Do not silently fill missing results with estimated scores.
- Calibrate model judges against human labels.
- Use application-specific evaluation before production rollout.
Current release tables are a useful source of candidate models, not a substitute for application tests. Terminal-Bench 4.0, Terminal-Bench 2.1, SWE-bench Verified and DeepSWE v1.1 are different evaluations. HLE-Verified is not the same measurement as HLE with tools.
Anthropic reports an uncertainty interval for its Opus 5.5 Terminal-Bench result and specifies safeguard routing. xAI marks one Grok result as high effort while neighboring columns use xhigh. DeepSeek distinguishes no-tools HLE from tool-enabled and text-only results. Copying only the largest percentage loses essential information.
“A benchmark is a measurement under conditions. Delete the conditions and you no longer have the measurement.”
For a product, maintain a fixed set of real inputs with privacy-reviewed expected outcomes. Include abstention, citation accuracy, valid tool actions and failure recovery. If an LLM judges outputs, calibrate it against human reviews rather than assuming that it is a reliable referee.
Publish uncertainty and costs, and keep tests in the release gate. This page does not claim a particular judge-agreement rate or benchmark win. The model index preserves sources and displays unknown values as missing data.
Field guide / 01
Choose measurements that match the failure cost
A customer-support assistant needs evidence support, policy compliance and correct escalation. A coding agent needs executable acceptance checks and patch review. A creative assistant may need a calibrated human preference rubric. Do not use the easiest metric merely because it is available.
Define critical failures that cannot be averaged away. One cross-user data leak should not disappear inside a high overall pass rate.
- → Attempted cases, accepted cases and skipped cases are separate counts.
- → Include failures in total cost.
- → Keep subjective grading separate from deterministic checks.
Field guide / 02
Keep uncertainty visible
Small samples give unstable percentages. Repeated generations are not necessarily independent, and closely related fixtures can exaggerate apparent confidence. Report sample size and test composition rather than drawing universal conclusions from a narrow suite.
The Lab analyzer reports simple rates and cost per accepted case from your labels. It does not calculate a confidence guarantee or declare a provider winner. Use the raw records to examine the failures that matter.
Before you ship
0/5Use this as a review aid. Checks reset when you leave this page.
Questions that come up
Does a 95% pass rate mean the app is safe?
Not by itself. It depends on what was tested, how it was scored and which failures remain. Safety-critical failures should be assessed separately.
Sources & verification
Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.
- 01Introducing Claude Opus 5.5Anthropic↗
- 02Introducing Grok 4.7xAI / SpaceXAI↗
- 03DeepSeek changelog — V4.1 Flash and V4 Pro correctionDeepSeek↗
- 04Introducing Gemini 3.8 Flash and Flash CyberGoogle↗
- 05promptfoo — evaluations and red teamingpromptfoo↗
- 06Arize Phoenix — tracing and evaluationArize AI↗