Your Trace Tree Is the API Your Evaluators Will Call

Names, types, and input/output are not cosmetics. Langfuse traces break silently when you rename the observation a judge already targets.

You will silently break every judge, dashboard, and experiment the day you clean up observation names you treated as labels. Names, types, and input/output are not cosmetics.

In this article: You will learn what a good Langfuse trace looks like, and why that shape is a contract, not a pretty tree. We walk the official best practices against Harbor Desk, a support-ticket assistant that classifies a ticket, retrieves a policy snippet, and drafts a reply. By the end you will treat observation names the way you treat URL paths: pick them, then keep them.

You finally have a tree. Classify, retrieve, draft. It looks like the work your app actually did. Then six weeks pass. An LLM-as-a-judge targets draft-reply by name. A dashboard aggregates cost on generations. An experiment compares the ticket text to the draft. Someone cleans up the names. The judge goes quiet. The dashboard goes empty. The experiment starts comparing kwargs blobs. Nothing throws.

That is the thing people miss about Langfuse traces. A tree is not a screenshot of a run. It is the API your evaluators, dashboards, and experiments will call. Names, types, and input/output are load-bearing. A rename is an API break.

This article is about making that tree still work next month.

The tree is already an API

Langfuse features do not look at your source. They look at the tree.

• LLM-as-a-judge evaluators target observations by name and type, and they read input and output.
• Dashboards filter and aggregate by trace and observation names.
• Dataset experiments compare trace input and output across runs.
• Saved views on the tracing table reference names and attributes.

Treat observation names the way you treat URL paths. Pick them. Then keep them.

A good Harbor Desk trace is five decisions, not fifty settings: one turn, the right types, a nested tree, stable names, and I/O a reviewer can read.

One Langfuse trace is one turn

Langfuse groups at three levels. Observations join a trace via trace_id. Traces join a session via session_id.

A trace is one self-contained unit of work. The docs give the good examples: one chatbot turn, one agent run, one pipeline execution. Harbor Desk's unit is one ticket message. The customer sends "the life jacket arrived soaked." You classify, retrieve, and draft. You stop. That is one trace.

If the customer replies "and I want store credit, not a refund," that is a second trace. You do not know when the thread ends, so you do not wait to close a mega-trace that holds the whole conversation. Sessions are how those two turns sit next to each other in a replay. A trace that tries to be the whole thread gets large, hard to navigate, and impossible to score turn by turn.

The UI shows the same trace as a tree and as an agent graph. Both views only help if the steps you care about are in the tree, typed correctly, and nested under the span that orchestrated them.

Audit the tree you already have

Open yesterday's Harbor Desk trace and ask the three questions the official best-practices page asks.

Are the right steps showing up, with the right type? The two LLM calls must be generations, or you lose tokens and cost. The policy lookup must be a retriever. The ticket handler is a span. Framework integrations set types for you. You set them with as_type in Python or asType in TypeScript when you instrument by hand.

Is it nested correctly? retrieve-policy and draft-reply sit under handle-ticket, as siblings of classify-intent. A tool or retriever dangling at the root means it ran outside the active context. Put the work back inside the with block or the startActiveObservation callback.

Is there noise you do not need? HTTP spans, database queries, and framework internals often clutter the tree without telling you what Harbor Desk did. If the tree is a thicket, filter the noisy instrumentation scopes. A dict lookup that you wrapped yourself will not produce those spans. The OpenAI drop-in and the Agent SDK instrumentor will.

Observation names are an API

Observation names show up in judges, dashboards, and the tracing table. When a name changes, every evaluator, dashboard query, and saved filter that targeted the old name silently stops matching.

Use active language. Verb first: classify-intent, retrieve-policy, and draft-reply. The tree then reads as a description of what the app did.

Keep dynamic values out of names. process-order, not process-order-4412 or draft-reply-retry-2. A name identifies the operation, not one execution of it. Put the order id and the retry count in metadata.

Do not name observations after the model, such as gpt-4o or claude-sonnet. If you swap the model next month, every filter breaks. The model is already an attribute on generation observations. Use that.

Harbor Desk's names already follow this. Keep them. If you feel the urge to clean them up when you port the app to an agent framework, resist it. The judge you write later will look for draft-reply.

Gotcha: A prettier name is not free. The next LLM-as-a-judge, cost chart, and saved view will miss the rows and will not tell you why.

Input and output a human can read

If an observation has neither input nor output, ask whether it should exist.

The root observation deserves the most care. Trace-level input and output are derived from it. They show up in the tracing table. They are what evaluators read, and they are what experiments compare. Set them to what a reviewer needs at a glance. For Harbor Desk, that is the ticket text in and the draft reply out, not a kwargs blob of function arguments. Put the raw payload in metadata if you still want it for debugging.

Typical I/O for a generation, from the docs: chatbot user message and assistant reply; RAG query and answer; and classification text and predicted label. Harbor Desk's classify-intent is the third: ticket in, refund out. draft-reply is the first: ticket-plus-policy in, the email draft out.

Most inputs and outputs can render as a role-labeled conversation when they are a list of {role, content} messages. If yours show up as raw JSON, check that format before you blame the UI.

The root already does this:

with langfuse.start_as_current_observation(
    as_type="span",
    name="handle-ticket",
    input={"ticket": TICKET},
) as root:
    # ...
    root.update(output={"intent": intent, "reply": reply})

That is the right instinct. If you later add a request id or the raw OpenAI response object, put those on metadata, not on the root I/O.

Attributes you will wish you had set

The rest of the best-practices list is a preview of labels, prompts, and scores. Name it so you do not invent a second system.

• Metadata is the key-value bag for evaluation context, request ids, retrieval stats, and raw payloads. Values are limited; oversized ones drop.
• Model, usage, and cost live on generations. The OpenAI drop-in fills them. Manual instrumentation has to fill them.
• Tags are business-level dimensions such as refund-flow. They are immutable, and you must set them at creation time. A judgment that you learn later is a score, not a tag.
• Prompt links attach a Langfuse prompt version to a generation.
• Environment, set to production, staging, or development, keeps test traces out of production dashboards.
• User id and session id answer "what did this customer see?" and "show me the thread."

You do not have to set all of these today. You do have to stop putting them in the observation name.

Do this today

Open yesterday's tree and treat it like a contract review. Four checks, then stop.

• Ask the three questions. Right types, correct nesting, leftover noise. Fix a missing generation or a retriever dangling at the root before you add anything new.
• Confirm the four names are verb-first and stable. handle-ticket, classify-intent, retrieve-policy, draft-reply. No order ids, no retry counts, no model names.
• Set the root observation's input to the ticket text and its output to the draft reply. Move kwargs and raw payloads to metadata.
• Try to rename draft-reply to something prettier. Then do not. The judge you write later will look for that name.

Boring on purpose

A good Harbor Desk trace is boring on purpose. One turn, four stable names, typed steps, ticket text in, and draft out. That boredom is what lets a judge, a dashboard, and an experiment target the same rows next month.

The tree you have today is already the API those features will call. Treat the names like URL paths. Pick them. Keep them. The day you clean them up is the day the evals go quiet, and nothing throws.

---