Practical guide. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.
For: Teams shipping prompts, agents or retrieval changes
Before you start: A model-backed task with an expected outcome · A small redacted fixture set
⚡ TL;DR — the 30-second version
- Begin with a small, human-understood regression set.
- Use deterministic assertions for deterministic requirements.
- Test the CI failure path and cap evaluation spend.
An evaluation suite should answer a narrow release question: does this proposed change improve the application without breaking an important behavior? promptfoo provides a CLI and comparison workflow for prompts, providers and tests. The useful work is deciding what counts as success.
Start with ten cases you can inspect by hand. Include normal inputs, one missing-field case, one refusal or permission boundary and at least one previous production failure. Keep these cases stable while you change a single part of the system.
“The best first eval is a regression you already understand well enough to score without another model.”
Field guide / 01
01 / Prefer direct checks where possible
Schema validity, allowed categories, required citations and forbidden side effects can often be checked deterministically. Use a model judge only for qualities that are difficult to express as code, and preserve a human-reviewed calibration set.
Do not give the judge the provider name or which candidate is supposed to be better. Randomize presentation when comparing prose, and inspect disagreements rather than treating a numeric judgment as a fact.
| Requirement | Suitable check | Watch out for |
|---|---|---|
| Valid output object | Schema validation | Valid structure, wrong values |
| Correct document cited | Allowed IDs + evidence check | Citation that does not support the claim |
| No unauthorized write | Tool trace inspection | Attempted versus executed actions |
| Helpful explanation | Calibrated rubric + human review | Judge preference for longer answers |
Field guide / 02
02 / Version the experiment
Record prompt hash, fixture version, model ID, provider settings and application commit. If retrieval changes, record the corpus version and index configuration. A result without these identifiers is hard to reproduce or explain later.
Keep failures in the dataset, but do not tune endlessly on the same small set. Add a held-out set for material changes and review whether your fixtures still resemble current user traffic.
Field guide / 03
03 / Make the gate fail honestly
The current CLI documentation distinguishes test failures from other errors and exposes exit-code behavior. Verify this in your own CI by intentionally breaking an assertion. A command that prints red text but exits successfully is not a release gate.
Store input, candidate output, assertion result, latency and usage as artifacts with sensitive data removed. Bound concurrency, requests and total spend; an accidental matrix expansion can become a costly incident.
Field guide / 04
04 / Report decisions, not just percentages
Summarize the number of attempted and accepted cases, the change in review burden and the total cost. Inspect the new failures individually. A one-point aggregate gain can hide the loss of a critical authorization check.
Use the Lab's result analyzer to summarize already-scored runs. It computes rates from supplied labels; it does not call a model or decide correctness for you. That makes the difference between measurement and judgment explicit.
Before you ship
0/5Use this as a review aid. Checks reset when you leave this page.
Sources & verification
Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.
- 01promptfoo — evaluations and red teamingpromptfoo↗
- 02promptfoo — CLI and CI exit behaviorpromptfoo↗
- 03Arize Phoenix — tracing and evaluationArize AI↗