The Playground Looked Fine. That Is Not a Langfuse Experiment.

An experiment is a task, a dataset, and evaluators. Local data is a smoke. A hosted dataset is a comparison. CI is the same runner with a threshold.

You already saw the French ticket come back in English. Shipping the next prompt change without running it against that ticket is how the same miss happens again.

In this article: You will learn how Langfuse experiments turn a live miss into a case the next prompt change must pass. We walk the building blocks (dataset, item, task, evaluation method, score, and experiment run), start with a local run_experiment smoke on Harbor Desk tickets, move the same cases onto a hosted dataset so runs compare in the UI, and put a threshold on that runner in CI so you do not promote harbor-desk-reply because the playground looked fine.

You changed the reply prompt. The playground sample looked warmer, still on-policy, still polite. You promoted the production label. The next French ticket came back in English.

Scoring live traces is how you notice that miss. A Langfuse experiment is how you make sure the next prompt change cannot ship it again. The runner is not a new product. It is Harbor Desk's task, a dataset of tickets, and the same scores you already know how to write.

Harbor Desk is a support-ticket assistant. A ticket comes in. It classifies the intent (refund, shipping, or other), retrieves a short policy snippet, and drafts a reply. The interesting failures live on that draft: a refund that never quotes the 30-day policy, a French shipping question answered in English. Those failures belong on a dataset before you touch the prompt again.

An experiment is a task, a dataset, and evaluators

An experiment runs a task against a dataset and evaluates the outputs. The building blocks, from the docs:

• Dataset: a collection of test cases.
• Dataset item: one case. Input (the ticket) plus optional expected output.
• Task: the application code under test. Harbor Desk's handle_ticket.
• Evaluation method: a code evaluator, a score you ingest, or an LLM-as-a-judge.
• Score: the output of that method.
• Experiment run: one execution of the task against every item.

run_experiment in Python and langfuse.experiment.run in JS/TS are the high-level runners. They handle concurrency, automatic tracing, item-level and run-level evaluators, and error isolation so one bad ticket does not kill the run.

That last clause is the reason you use the runner instead of a for loop in a notebook. A flaky classify on item three should show up as a failed item, not as an aborted script that never scored items four through twenty.

Start local, then move the dataset into Langfuse

Local data is the smallest proof. You get traces and scores. You do not get a dataset run in the UI.

This local smoke is three pieces: a sync task that wraps handle_ticket, a list of ticket dicts, and one run_experiment call.

import asyncio

from langfuse import get_client
from langfuse.openai import OpenAI

langfuse = get_client()


def harbor_desk_task(*, item, **kwargs):  # ①
    ticket = item["input"]  # ②
    # Part 5's handle_ticket is async and returns the draft.
    # The experiment runner's documented task is a sync function.
    return asyncio.run(handle_ticket(ticket))  # ③


local_data = [  # ④
    {
        "input": "The life jacket arrived soaked. I want a refund.",
        "expected_output": "A refund reply that quotes the 30-day damaged-goods policy.",
    },
    {
        "input": "Où est mon colis?",
        "expected_output": "A shipping reply written in French, not English.",
    },
]

result = langfuse.run_experiment(  # ⑤
    name="Harbor Desk tickets",
    description="Local smoke of classify-and-reply",
    data=local_data,  # ⑥
    task=harbor_desk_task,
)
print(result.format())

① The runner's documented task is a keyword-only function: item plus **kwargs. ② On local dict items, the ticket is item["input"]. A hosted DatasetItem uses item.input instead. ③ handle_ticket is async (Part 5). The runner expects a sync function, so asyncio.run bridges the two. ④ Two in-memory cases: a refund ticket and the French shipping miss from Part 7. ⑤ run_experiment is the high-level runner: concurrency, tracing, and error isolation. ⑥ data= is what makes this local. A hosted dataset drops this argument and calls dataset.run_experiment instead.

Note: The full extracted listing at code/langfuse/part-8-datasets-and-experiments/listings/01-local-run-experiment.py is the complete listing shown here.

The task signature is *, item, **kwargs. On a Langfuse-hosted dataset, item is a DatasetItem and the ticket is item.input. Write the task so it can read both shapes, or keep a thin adapter. Do not discover that difference on the first hosted run.

JS is the same idea. Set up LangfuseSpanProcessor first. ExperimentTask receives the item. Shut the OTEL SDK down when the script ends.

const result = await langfuse.experiment.run({
  name: "Harbor Desk tickets",
  description: "Local smoke of classify-and-reply",
  data: localData,
  task: myTask,
});
console.log(await result.format());
await otelSdk.shutdown();

When the cases live in Langfuse, you get a dataset run you can compare in the UI:

dataset = langfuse.get_dataset("harbor-desk-tickets")
result = dataset.run_experiment(
    name="Reply prompt v4",
    description="Less formal wording, still policy-faithful",
    task=harbor_desk_task,
)
print(result.format())

const dataset = await langfuse.dataset.get("harbor-desk-tickets");
const result = await dataset.runExperiment({
  name: "Reply prompt v4",
  task: myTask,
});

Name the run after the change you are testing (Reply prompt v4), not after the dataset. The dataset is the fixture. The run is the candidate. Side-by-side comparison in the UI only helps if those names stay stable and specific.

Add the French ticket as an item whose expected output is "reply in the customer's language," not a classify label. That is the whole online-to-offline loop: a live language miss becomes a case the next experiment must pass. The first item freezes the refund-policy reply. Neither item is a classify-only smoke.

CI is the same runner plus a threshold

The CI workflow from the docs:

1. Keep a Langfuse dataset of Harbor Desk tickets.
2. Write an experiment with the SDK.
3. Attach evaluators: the boolean mentions-policy check and a judge on draft-reply.
4. Raise RegressionError when a score crosses a line.
5. Run it from GitHub Actions with langfuse/experiment-action.

The action needs Python SDK v4.6.0+ or JS SDK v5.3.0+. Pin the action to a release. The default trigger in the cookbook is pull_request.

That is the whole gate. You already have the dataset. You already have the task. You already have the scores. CI adds a number you are willing to fail a pull request over. If mentions-policy drops, or the draft-reply judge says the French item came back in English, the promote does not happen.

In production: offline experiments decide whether to promote the production label on harbor-desk-reply. Online judges watch what you already shipped. Do not skip the gate because the UI playground "looked fine."

Do this today

• Capture the miss. Take one live failure (a French ticket that got an English reply is the Harbor Desk version) and write it as a dataset item whose expected output names the behavior, not a classify label.
• Freeze a policy case. Add a refund ticket whose expected output is a reply that quotes the damaged-goods policy. Neither item should be a classify-only smoke.
• Run a local smoke. Wrap handle_ticket as a keyword-only sync task and call langfuse.run_experiment(...) (or langfuse.experiment.run in JS) on that list. Print result.format().
• Move the cases into harbor-desk-tickets. Re-run with dataset.run_experiment / dataset.runExperiment so the UI can compare this prompt version to the last one.
• Put a threshold on it. Attach mentions-policy and a judge on draft-reply, raise RegressionError when a score crosses a line, and wire langfuse/experiment-action on pull_request. Pin the action and the SDK (v4.6.0+ Python, v5.3.0+ JS).

The playground is not the gate

An experiment is not a new product. It is Harbor Desk's task, a dataset of tickets, and the same scores you already know how to write. Local data is for a smoke. A hosted dataset is for comparison. CI is the smoke with a threshold.

The French miss belongs in harbor-desk-tickets before you touch the prompt again. Offline experiments decide whether to promote. Online judges watch what you already shipped. The playground can still look fine. That is not the argument. The argument is whether the next production label on harbor-desk-reply survived the cases you already lost in production.

If it did not run against the dataset, it is not ready to ship.

---