Practical guide. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.
For: Developers operating LLM applications
Before you start: A request or job identifier · An inventory of sensitive data
⚡ TL;DR — the 30-second version
- Trace steps and decisions, not only final text.
- Redact before telemetry leaves the application.
- Treat trace stores and evaluation datasets as sensitive systems.
When an answer is wrong, you need to know whether the failure came from retrieval, the model, a tool, permissions or a downstream consumer. A single log line containing the final response cannot explain that chain. Structured traces can—but they can also copy private documents and credentials into a second system.
Langfuse and Phoenix both document model-aware tracing and evaluation workflows. Choose an observability backend after defining the information you actually need and the data you are not allowed to retain.
“A perfect trace of an unsafe data flow is still an unsafe data flow. Redaction happens before export.”
Field guide / 01
01 / Trace the user-visible job
Use one trace for a complete user job, with child spans for retrieval, model requests, tool calls, validation and writes. Connect retries to their original operation. A latency number is useful only when you can distinguish queueing, inference, tools and review.
Record model ID, prompt version, fixture or corpus version and final outcome. These identifiers let you compare deployments without retaining every raw prompt.
{
"job_id": "opaque-internal-id",
"model_id": "pinned-model-id",
"prompt_version": "support-v3",
"outcome": "review_required",
"usage": { "input": 12000, "cached": 8000, "output": 600 },
"redaction_policy": "support-minimal-v1"
}Field guide / 02
02 / Redact at the producer
Do not rely on a dashboard to hide data already transmitted. Remove credentials, customer identifiers and unnecessary document text before sending telemetry. Store a controlled reference to the original material when a reviewer genuinely needs to inspect it.
The Langfuse SDK documentation says tracing credentials belong on the server. Exposing a secret key in a web app is not fixed by calling it an analytics key. Apply the same standard to evaluator traces and debugging exports.
Field guide / 03
03 / Separate usage from quality
A successful HTTP request means the provider returned a response. It says nothing about whether the answer was correct or a proposed action was permitted. Track validation outcomes, review decisions and task acceptance as separate fields.
Cost estimates should preserve the units the provider bills: ordinary input, cache reads, cache writes, output, searches or audio minutes. Reconcile estimates periodically with actual invoices and mark unknown charges rather than filling them with zero.
Field guide / 04
04 / Test deletion and access
Choose retention by data category and operational need. Test whether deleting a customer or document removes related raw trace content, evaluation copies and exports. Restrict who can read production traces, not just who can administer the dashboard.
Promote only privacy-reviewed examples into long-lived regression datasets. A failure can be worth learning from without keeping the customer's entire conversation forever.
Before you ship
0/5Use this as a review aid. Checks reset when you leave this page.
Sources & verification
Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.
- 01Langfuse — LLM observabilityLangfuse↗
- 02Langfuse — server-side SDKs and secret handlingLangfuse↗
- 03Arize Phoenix — tracing and evaluationArize AI↗
- 04OpenAI — current prompt-caching behaviorOpenAI↗