Evaluation design. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.
For: Developers and teams putting AI into practical use
⚡ TL;DR — the 30-second version
- Freeze the repository and evaluation tasks.
- Log model, harness, effort, tools and budget.
- Include failures and human review in the cost.
- Publish results only when the actual traces and methodology exist.
Start with a frozen repository and a written task set. Include tasks you already understand well enough to grade: a bug with a hidden regression test, a bounded feature, a migration and a cross-file refactor. Remove secrets and third-party customer data before exposing a repository to an agent.
Record the exact model ID, harness version, effort setting, available tools and time/token budget. Current release tables use different benchmarks and different effort levels; a Terminal-Bench 4.0 number cannot replace a SWE-bench Verified number. Even the same benchmark can differ with the harness and sampling policy.
“Autonomy is not the outcome. A reviewed, correct change is the outcome—and your test should measure exactly that.”
Use hidden or independently maintained tests. Require a human diff review for test changes and irreversible operations. Capture total input, cached input, output, tool cost and review time for every run—including failures—then report cost per accepted change rather than cost per successful-looking demo.
Repeat tasks where practical and publish variance. A narrow lead from a small sample should not become a universal winner. This page describes the method; it does not claim a completed DARC test, published traces or a real nine-agent ranking.
Field guide / 01
Write a task card before running an agent
A useful task card contains the repository commit, user-visible problem, allowed files, prohibited changes, acceptance tests and maximum budget. Include a stop condition for repeated failure or uncertain destructive actions.
Separate visible development tests from the final acceptance checks. An agent can legitimately edit tests during development, but the final judge must not be a test suite whose assertions were weakened to make the patch pass.
| Record | Example requirement |
|---|---|
| Correctness | All independently maintained acceptance cases pass |
| Scope | No unrelated dependency or permission changes |
| Verification | Every claimed command is present in the trace |
| Cost | All attempts counted, including abandoned runs |
Field guide / 02
Report the distribution, not the best demo
Repeat tasks when sampling variance matters and report the number of attempts. Show failures and intervention counts beside the success rate. A single accepted patch on an easy task does not establish reliability on a production backlog.
The Lab analyzer can summarize supplied pass/fail labels and cost values. Its example rows are intentionally named Candidate A and Candidate B so no illustrative dataset can be mistaken for a real provider benchmark.
Before you ship
0/5Use this as a review aid. Checks reset when you leave this page.
Questions that come up
Can an LLM be the only judge of code correctness?
Not safely as a general rule. Use executable checks and human review for behavior and security; a model critique can supplement those checks but should not replace them.
Sources & verification
Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.
- 01Introducing Claude Opus 5.5Anthropic↗
- 02Introducing Grok 4.7xAI / SpaceXAI↗
- 03DeepSeek changelog — V4.1 Flash and V4 Pro correctionDeepSeek↗
- 04promptfoo — evaluations and red teamingpromptfoo↗
