DROPGoogle launches Gemini 3.8 Flash TTS and Flash-Lite TTS///DROPGemini Omni brings 1080p generation to Google Vids///BREAKClaude Opus 5.5 is here: 1M context, $4 input / $20 output///BREAKOpenAI adds GPT-6 Sol and Luna to the GPT-6 family///PAPEROpus 5.5's system card documents gains—and evaluation limits///BREAKGrok 4.7 launches at $2 input / $6 output per million tokens///DROPGrok Voice Transcribe 2.0 targets difficult real-world audio///TRENDGoogle steers new Gemini API projects to 3.8 Flash or 3.5 Flash-Lite///DROPGoogle launches Gemini 3.8 Flash TTS and Flash-Lite TTS///DROPGemini Omni brings 1080p generation to Google Vids///BREAKClaude Opus 5.5 is here: 1M context, $4 input / $20 output///BREAKOpenAI adds GPT-6 Sol and Luna to the GPT-6 family///PAPEROpus 5.5's system card documents gains—and evaluation limits///BREAKGrok 4.7 launches at $2 input / $6 output per million tokens///DROPGrok Voice Transcribe 2.0 targets difficult real-world audio///TRENDGoogle steers new Gemini API projects to 3.8 Flash or 3.5 Flash-Lite///
DARC.DEV
All guides
MethodologyLevel · Build#Agents#Coding#Evals

How to benchmark coding agents without gaming the result

Design a reproducible evaluation on a real repository: task cards, protected acceptance tests, fixed budgets and cost per reviewed change.

DCDariusz Czechowicz23 Sept 20263 min readEvaluation design
How to benchmark coding agents without gaming the result
Source-checked Sources & corrections
2 practical sections · checklist · sources

Evaluation design. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.

For: Developers and teams putting AI into practical use

⚡ TL;DR — the 30-second version

  • Freeze the repository and evaluation tasks.
  • Log model, harness, effort, tools and budget.
  • Include failures and human review in the cost.
  • Publish results only when the actual traces and methodology exist.

Start with a frozen repository and a written task set. Include tasks you already understand well enough to grade: a bug with a hidden regression test, a bounded feature, a migration and a cross-file refactor. Remove secrets and third-party customer data before exposing a repository to an agent.

Record the exact model ID, harness version, effort setting, available tools and time/token budget. Current release tables use different benchmarks and different effort levels; a Terminal-Bench 4.0 number cannot replace a SWE-bench Verified number. Even the same benchmark can differ with the harness and sampling policy.

“Autonomy is not the outcome. A reviewed, correct change is the outcome—and your test should measure exactly that.”
— Editorial interpretation

Use hidden or independently maintained tests. Require a human diff review for test changes and irreversible operations. Capture total input, cached input, output, tool cost and review time for every run—including failures—then report cost per accepted change rather than cost per successful-looking demo.

Repeat tasks where practical and publish variance. A narrow lead from a small sample should not become a universal winner. This page describes the method; it does not claim a completed DARC test, published traces or a real nine-agent ranking.

Field guide / 01

Write a task card before running an agent

A useful task card contains the repository commit, user-visible problem, allowed files, prohibited changes, acceptance tests and maximum budget. Include a stop condition for repeated failure or uncertain destructive actions.

Separate visible development tests from the final acceptance checks. An agent can legitimately edit tests during development, but the final judge must not be a test suite whose assertions were weakened to make the patch pass.

RecordExample requirement
CorrectnessAll independently maintained acceptance cases pass
ScopeNo unrelated dependency or permission changes
VerificationEvery claimed command is present in the trace
CostAll attempts counted, including abandoned runs

Field guide / 02

Report the distribution, not the best demo

Repeat tasks when sampling variance matters and report the number of attempts. Show failures and intervention counts beside the success rate. A single accepted patch on an easy task does not establish reliability on a production backlog.

The Lab analyzer can summarize supplied pass/fail labels and cost values. Its example rows are intentionally named Candidate A and Candidate B so no illustrative dataset can be mistaken for a real provider benchmark.

Before you ship

0/5

Use this as a review aid. Checks reset when you leave this page.

Questions that come up

Can an LLM be the only judge of code correctness?

Not safely as a general rule. Use executable checks and human review for behavior and security; a model critique can supplement those checks but should not replace them.

Put it into practiceUse the result analyzer

Sources & verification

Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.

  1. 01Introducing Claude Opus 5.5Anthropic↗
  2. 02Introducing Grok 4.7xAI / SpaceXAI↗
  3. 03DeepSeek changelog — V4.1 Flash and V4 Pro correctionDeepSeek↗
  4. 04promptfoo — evaluations and red teamingpromptfoo↗
Editorial policy and correction log →
#Agents#Coding#Evals