DROPGoogle launches Gemini 3.8 Flash TTS and Flash-Lite TTS///DROPGemini Omni brings 1080p generation to Google Vids///BREAKClaude Opus 5.5 is here: 1M context, $4 input / $20 output///BREAKOpenAI adds GPT-6 Sol and Luna to the GPT-6 family///PAPEROpus 5.5's system card documents gains—and evaluation limits///BREAKGrok 4.7 launches at $2 input / $6 output per million tokens///DROPGrok Voice Transcribe 2.0 targets difficult real-world audio///TRENDGoogle steers new Gemini API projects to 3.8 Flash or 3.5 Flash-Lite///DROPGoogle launches Gemini 3.8 Flash TTS and Flash-Lite TTS///DROPGemini Omni brings 1080p generation to Google Vids///BREAKClaude Opus 5.5 is here: 1M context, $4 input / $20 output///BREAKOpenAI adds GPT-6 Sol and Luna to the GPT-6 family///PAPEROpus 5.5's system card documents gains—and evaluation limits///BREAKGrok 4.7 launches at $2 input / $6 output per million tokens///DROPGrok Voice Transcribe 2.0 targets difficult real-world audio///TRENDGoogle steers new Gemini API projects to 3.8 Flash or 3.5 Flash-Lite///
DARC.DEV
All guides
GuideLevel · Deep#Evals#Production#Quality

Evals: separate a published score from your product's reliability

Benchmark version, harness, effort and uncertainty: the minimum evidence needed before turning a number into a deployment decision.

DCDariusz Czechowicz23 Sept 20263 min readEvaluation design
10
Guide
Source-checked Sources & corrections
2 practical sections · checklist · sources

Evaluation design. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.

For: Developers and teams putting AI into practical use

⚡ TL;DR — the 30-second version

  • Compare the exact benchmark version and run conditions.
  • Do not silently fill missing results with estimated scores.
  • Calibrate model judges against human labels.
  • Use application-specific evaluation before production rollout.

Current release tables are a useful source of candidate models, not a substitute for application tests. Terminal-Bench 4.0, Terminal-Bench 2.1, SWE-bench Verified and DeepSWE v1.1 are different evaluations. HLE-Verified is not the same measurement as HLE with tools.

Anthropic reports an uncertainty interval for its Opus 5.5 Terminal-Bench result and specifies safeguard routing. xAI marks one Grok result as high effort while neighboring columns use xhigh. DeepSeek distinguishes no-tools HLE from tool-enabled and text-only results. Copying only the largest percentage loses essential information.

“A benchmark is a measurement under conditions. Delete the conditions and you no longer have the measurement.”
— Editorial interpretation

For a product, maintain a fixed set of real inputs with privacy-reviewed expected outcomes. Include abstention, citation accuracy, valid tool actions and failure recovery. If an LLM judges outputs, calibrate it against human reviews rather than assuming that it is a reliable referee.

Publish uncertainty and costs, and keep tests in the release gate. This page does not claim a particular judge-agreement rate or benchmark win. The model index preserves sources and displays unknown values as missing data.

Field guide / 01

Choose measurements that match the failure cost

A customer-support assistant needs evidence support, policy compliance and correct escalation. A coding agent needs executable acceptance checks and patch review. A creative assistant may need a calibrated human preference rubric. Do not use the easiest metric merely because it is available.

Define critical failures that cannot be averaged away. One cross-user data leak should not disappear inside a high overall pass rate.

  • → Attempted cases, accepted cases and skipped cases are separate counts.
  • → Include failures in total cost.
  • → Keep subjective grading separate from deterministic checks.

Field guide / 02

Keep uncertainty visible

Small samples give unstable percentages. Repeated generations are not necessarily independent, and closely related fixtures can exaggerate apparent confidence. Report sample size and test composition rather than drawing universal conclusions from a narrow suite.

The Lab analyzer reports simple rates and cost per accepted case from your labels. It does not calculate a confidence guarantee or declare a provider winner. Use the raw records to examine the failures that matter.

Before you ship

0/5

Use this as a review aid. Checks reset when you leave this page.

Questions that come up

Does a 95% pass rate mean the app is safe?

Not by itself. It depends on what was tested, how it was scored and which failures remain. Safety-critical failures should be assessed separately.

Put it into practiceAnalyze a labeled run

Sources & verification

Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.

  1. 01Introducing Claude Opus 5.5Anthropic↗
  2. 02Introducing Grok 4.7xAI / SpaceXAI↗
  3. 03DeepSeek changelog — V4.1 Flash and V4 Pro correctionDeepSeek↗
  4. 04Introducing Gemini 3.8 Flash and Flash CyberGoogle↗
  5. 05promptfoo — evaluations and red teamingpromptfoo↗
  6. 06Arize Phoenix — tracing and evaluationArize AI↗
Editorial policy and correction log →
#Evals#Production#Quality