DROPGoogle launches Gemini 3.8 Flash TTS and Flash-Lite TTS///DROPGemini Omni brings 1080p generation to Google Vids///BREAKClaude Opus 5.5 is here: 1M context, $4 input / $20 output///BREAKOpenAI adds GPT-6 Sol and Luna to the GPT-6 family///PAPEROpus 5.5's system card documents gains—and evaluation limits///BREAKGrok 4.7 launches at $2 input / $6 output per million tokens///DROPGrok Voice Transcribe 2.0 targets difficult real-world audio///TRENDGoogle steers new Gemini API projects to 3.8 Flash or 3.5 Flash-Lite///DROPGoogle launches Gemini 3.8 Flash TTS and Flash-Lite TTS///DROPGemini Omni brings 1080p generation to Google Vids///BREAKClaude Opus 5.5 is here: 1M context, $4 input / $20 output///BREAKOpenAI adds GPT-6 Sol and Luna to the GPT-6 family///PAPEROpus 5.5's system card documents gains—and evaluation limits///BREAKGrok 4.7 launches at $2 input / $6 output per million tokens///DROPGrok Voice Transcribe 2.0 targets difficult real-world audio///TRENDGoogle steers new Gemini API projects to 3.8 Flash or 3.5 Flash-Lite///
DARC.DEV
All guides
TutorialLevel · Build#Evals#CI#promptfoo

Put prompt changes behind a test gate, not a vibes check

A practical promptfoo workflow: fixtures, deterministic assertions, calibrated judges, budget caps and CI artifacts you can actually debug.

DCDariusz Czechowicz23 Sept 20263 min readPractical guide
69
Tutorial
Source-checked Sources & corrections
4 practical sections · checklist · sources

Practical guide. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.

For: Teams shipping prompts, agents or retrieval changes

Before you start: A model-backed task with an expected outcome · A small redacted fixture set

⚡ TL;DR — the 30-second version

  • Begin with a small, human-understood regression set.
  • Use deterministic assertions for deterministic requirements.
  • Test the CI failure path and cap evaluation spend.

An evaluation suite should answer a narrow release question: does this proposed change improve the application without breaking an important behavior? promptfoo provides a CLI and comparison workflow for prompts, providers and tests. The useful work is deciding what counts as success.

Start with ten cases you can inspect by hand. Include normal inputs, one missing-field case, one refusal or permission boundary and at least one previous production failure. Keep these cases stable while you change a single part of the system.

“The best first eval is a regression you already understand well enough to score without another model.”
— Editorial interpretation

Field guide / 01

01 / Prefer direct checks where possible

Schema validity, allowed categories, required citations and forbidden side effects can often be checked deterministically. Use a model judge only for qualities that are difficult to express as code, and preserve a human-reviewed calibration set.

Do not give the judge the provider name or which candidate is supposed to be better. Randomize presentation when comparing prose, and inspect disagreements rather than treating a numeric judgment as a fact.

RequirementSuitable checkWatch out for
Valid output objectSchema validationValid structure, wrong values
Correct document citedAllowed IDs + evidence checkCitation that does not support the claim
No unauthorized writeTool trace inspectionAttempted versus executed actions
Helpful explanationCalibrated rubric + human reviewJudge preference for longer answers

Field guide / 02

02 / Version the experiment

Record prompt hash, fixture version, model ID, provider settings and application commit. If retrieval changes, record the corpus version and index configuration. A result without these identifiers is hard to reproduce or explain later.

Keep failures in the dataset, but do not tune endlessly on the same small set. Add a held-out set for material changes and review whether your fixtures still resemble current user traffic.

Field guide / 03

03 / Make the gate fail honestly

The current CLI documentation distinguishes test failures from other errors and exposes exit-code behavior. Verify this in your own CI by intentionally breaking an assertion. A command that prints red text but exits successfully is not a release gate.

Store input, candidate output, assertion result, latency and usage as artifacts with sensitive data removed. Bound concurrency, requests and total spend; an accidental matrix expansion can become a costly incident.

Field guide / 04

04 / Report decisions, not just percentages

Summarize the number of attempted and accepted cases, the change in review burden and the total cost. Inspect the new failures individually. A one-point aggregate gain can hide the loss of a critical authorization check.

Use the Lab's result analyzer to summarize already-scored runs. It computes rates from supplied labels; it does not call a model or decide correctness for you. That makes the difference between measurement and judgment explicit.

Before you ship

0/5

Use this as a review aid. Checks reset when you leave this page.

Put it into practiceAnalyze labeled evaluation results

Sources & verification

Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.

  1. 01promptfoo — evaluations and red teamingpromptfoo↗
  2. 02promptfoo — CLI and CI exit behaviorpromptfoo↗
  3. 03Arize Phoenix — tracing and evaluationArize AI↗
Editorial policy and correction log →
#Evals#CI#promptfoo