Evaluation design. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.
For: Developers and teams putting AI into practical use
⚡ TL;DR — the 30-second version
- Use externally checkable escalation criteria.
- Measure accepted-task cost and retries.
- Treat published token rates as inputs, not completed-task economics.
Split workloads by risk and verifiability. Extraction with a strict schema is easier to validate automatically than open-ended advice. Define escalation criteria that are checked by your application, rather than trusting an uncalibrated model confidence claim.
Current prices suggest useful candidates, not automatic winners: GPT-6 Luna targets high-volume tasks, Sol offers another capability tier, and DeepSeek's peak/off-peak rates can change a batch workload's economics. More expensive models can still cost less per accepted result when retries fall.
“Cost reduction is only a win if the work still meets its acceptance criteria.”
Measure output quality, retries, review time, cached tokens and all billing categories on the same fixed task set. Keep a direct-to-frontier control group and record the quality/cost trade-off instead of reporting savings alone.
Use the Token Thermostat cost workbench to explore a bounded text workload before running it. Its output is an estimate from stated inputs, not an invoice or a prediction of quality. Use the result analyzer once you have actual labeled runs.
Field guide / 01
Use cost per accepted task
Sum model, tool and retry charges for all attempts, then divide by the number of accepted results. Do not remove failed runs from the numerator. If no task is accepted, the ratio is undefined—not zero.
Keep human review time alongside the invoice figure. A lower token bill can simply move work from the model to an operator, which may or may not be the right trade-off.
Field guide / 02
Test routing as a separate policy
Freeze the model and prompt configurations, then compare direct-to-frontier work with a bounded escalation rule. Log which condition caused escalation and whether the final result improved. Model self-confidence is not enough unless it has been calibrated.
Include peak/off-peak pricing, context thresholds and cache behavior where applicable. The Lab's estimator exposes input assumptions and exclusions so a scenario estimate is not mistaken for a provider invoice.
Before you ship
0/5Use this as a review aid. Checks reset when you leave this page.
Questions that come up
Is the cheapest token rate the cheapest workflow?
No. Output length, retries, cache reuse, tools and review effort determine the total. Compare accepted-task cost with a quality threshold.
Sources & verification
Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.
- 01GPT-6 Luna API model and pricingOpenAI↗
- 02GPT-6 Sol API model and pricingOpenAI↗
- 03DeepSeek models and pricingDeepSeek↗
- 04Introducing Claude Opus 5.5Anthropic↗
- 05LiteLLM — budgets, limits and virtual keysLiteLLM↗
- 06OpenAI — current prompt-caching behaviorOpenAI↗