You Changed the Prompt Without a Deploy. You Still Cannot Tell If the Reply Is Any Good.

Cover image for “You Changed the Prompt Without a Deploy. You Still Cannot Tell If the Reply Is Any Good.” by Rick Hightower

Langfuse scores turn thumbs, judges, and cheap Python checks into one named object on the live trace. Online evaluation starts here.

You can change the prompt without a deploy, and you still have no idea whether the draft is any good. Langfuse scores turn thumbs, judges, and cheap Python checks into one named object on the live trace.

In this article: You will learn what a Langfuse score actually is: a name, a value, and a data type, written by every evaluation method onto the same object. We cover the four types, the three Python writers, the order a team actually adopts the other methods, and the online versus offline split. By the end you will put a boolean policy check on one Harbor Desk run and a thumbs-down on a draft you do not like.

Harbor Desk now drafts a reply from a prompt you can change without a deploy. That is the win you wanted. It is also a new kind of blindness. Product can iterate on wording overnight, and you still do not know whether the latest draft is any good.

Harbor Desk is a support-ticket assistant. A ticket comes in. It classifies the intent, retrieves a short policy snippet, and drafts a reply. A Langfuse score is the object for judging that draft: a name, a value, and a data type (NUMERIC, CATEGORICAL, BOOLEAN, or TEXT). Scores attach to traces, observations, sessions, or dataset runs. Thumbs, judges, annotation queues, UI clicks, code evaluators, and create_score all write it.

This is online evaluation: scoring live traces. Offline evaluation, a dataset plus an experiment, is the next move. The loop the docs describe is the point. You score production, interesting failures become dataset items, you experiment, you promote, and you score again.

The Langfuse call does not care whether the tree came from the OpenAI drop-in, the Agent SDK instrumentor, or the DeepAgents callback. You need a trace_id and, optionally, an observation_id.

Every evaluation method writes the same Langfuse score object, which then attaches to a trace, an observation, a session, or a dataset run.

Four types, four jobs

NUMERIC is a float. Helpfulness from 0 to 1.
CATEGORICAL is a label. correct, partially_correct, and incorrect.
BOOLEAN is yes or no. Did the draft mention the policy? Did the user disagree?
TEXT is an open-ended note.

Use numeric for a scale, categorical for a closed set of labels, boolean for a gate, and text for a reviewer comment you do not want to force into a number.

A mindmap of the four Langfuse score types and the job each one is for: a numeric scale, a closed set of labels, a yes-or-no gate, and an open-ended note.

Write a score from your app

There are three Python entry points, from the scores-via-SDK page.

The same score object can be written after the fact from stored ids, or from inside the span that produced the draft. span.score attaches to the current observation; span.score_trace attaches to the whole trace.

from langfuse import get_client

langfuse = get_client()

# 1. You have ids (from a worker, a webhook, a nightly job)
langfuse.create_score(  # ①
    name="mentions-policy",
    value=1.0,
    trace_id=trace_id,
    observation_id=observation_id,  # optional; include both if you target a step  ②
    data_type="BOOLEAN",
    comment="Draft quoted the 30-day refund policy",
)

# 2. You are inside the span. handle_ticket is the async
#    function from Part 5; it returns the draft string.
with langfuse.start_as_current_observation(as_type="span", name="handle-ticket") as span:  # ③
    reply = await handle_ticket(ticket)
    span.score(  # ④
        name="mentions-policy",
        value=float("30 days" in reply.lower()),
        data_type="BOOLEAN",
    )
    span.score_trace(  # ⑤
        name="overall_quality",
        value=0.8,
        data_type="NUMERIC",
    )

① create_score writes the judgment after the fact, using ids a worker, webhook, or nightly job already stored. ② Include observation_id with trace_id to pin the score to one step; omit it and the score lands on the trace. ③ The second call site is inside the handle-ticket span, so the client already knows which observation is current. ④ span.score attaches the boolean policy check to that current observation. ⑤ span.score_trace writes a numeric quality score on the whole trace, not the inner span.

Note: The full extracted listing at code/langfuse/part-7-scores-and-evaluation/listings/01-create-score-and-span-score.py shows the parts elided here.

create_score can land before the trace exists. It shows up in the scores table and links when the matching trace_id arrives. If you attach a score to an observation, send both the observation id and the trace id.

A cheap Harbor Desk check: does the draft contain a phrase from the policy that you retrieved? That is a boolean you can compute without another model. Put it on draft-reply if you have that observation id, or on the trace if you do not.

Two ways to write a Langfuse score: a nightly job calls create_score with stored ids, and Harbor Desk scores from inside the handle-ticket span.

The methods a team actually adopts

Teams do not start with a judge. They start with a click.

Scores via the UI. Click a trace, add a score. Fine for spot checks. Not a pipeline.

Annotation queues. Structured human review. Use them to build ground truth, not to grade every production ticket by hand.

User feedback. Maya hits thumbs-down on the reply. That is a score on the live trace. The docs treat this as first-class, not a side channel.

Code evaluators. Deterministic Python or TypeScript Langfuse runs for you. Use them when the check is a rule, such as JSON parsed or policy id present, and you want it authored in the UI and reused on observations and experiments.

LLM-as-a-judge. Another model scores the output against a rubric. Observation-level judges are what you want on live data. Trace-level judges are deprecated. You target by observation name and type, which is why draft-reply is an API, not a label you can rename on a whim. A typical judge prompt has criteria, the ticket, the draft, and optionally a reference answer. The judge writes a score plus reasoning. Numeric for helpfulness, categorical for correct / partial, and boolean for policy violations.

You configure the judge in the Langfuse UI or via the API. You do not have to stand up a second model caller in Harbor Desk to get started. Point it at draft-reply generations. If you renamed that observation and the judge is silent, that is the silent API break: judges target by name and type.

Gotcha: tags are immutable. You must set them at creation. A judgment that you learn after the fact is a score. Do not try to tag a trace unhelpful from a judge.

The order a Harbor Desk team actually adopts scoring: UI clicks, annotation queues, user thumbs, code evaluators, then an LLM-as-a-judge pointed at draft-reply.

Online scores live traces

Online scores live traces. That is this article. Catch the French ticket that got an English reply.

Offline runs your task against a fixed dataset before you promote a prompt.

The docs' support-chatbot story is Harbor Desk's story. You change the reply prompt to be less formal and experiment on a dataset first. You promote, watch online scores, add the French miss to the dataset, and experiment again.

The online and offline loop: score live traces, turn interesting failures into dataset items, experiment, promote, and score production again.

Do this today

Add the boolean mentions-policy check to one Harbor Desk run. Use span.score inside handle-ticket, or create_score if you already have the ids. The check is a string search for a policy phrase, not another model call.
Put that score on draft-reply if you have the observation id. Otherwise put it on the trace. Send both ids when you target a step.
Click a thumbs-down on a draft you do not like. Confirm it lands as a score on the live trace, not as a tag.
Point an observation-level judge at draft-reply generations if you want a helpfulness number. Do not start a second model caller in the app. Do not use a deprecated trace-level judge.
Do not tag a trace unhelpful after the fact. Tags are immutable. After-the-fact judgment is a score.

One object, many writers

A score is a named judgment. create_score, span.score, and span.score_trace write them from code. Judges, code evaluators, annotation queues, UI clicks, and thumbs write the same object. Target draft-reply by name. Online is production. Offline is the experiment you run before the next prompt promote.

You pulled the prompt out of the repo so wording could move without a deploy. A score is how you find out whether that wording is any good, on the same tree, without inventing a second system.

Add the boolean. Click one thumbs-down. Then turn the failures into a dataset and run the experiment before you promote again.