Integration · Langfuse
rightmodeler + Langfuse
Langfuse already records what every generation cost and produced. rightmodeler replays those observations through cheaper models and reports agreement, evidence, sample size, and abstentions step by step, scored by Langfuse itself when you point it there.
TL;DR
rightmodeler has a dedicated Langfuse adapter. It recognizes observation exports by their trace_id, parent_observation_id, and type keys, maps the observations whose type is GENERATION, and reads provided_model_name, input.messages, output, tool calls, and usage_details per record. Langfuse can also be the evaluator: pass --evaluator langfuse and your own scorers grade the replays, with the built-in judge used only when nothing better is reachable.
How it works with Langfuse
Export observations to a file
Pull observations from GET /api/public/observations (basic auth with your LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY) and save the data array as JSON, or take a JSONL batch export from blob storage. A file on disk is all the CLI needs.
Autodetected, then mapped field by field
The adapter recognizes Langfuse by trace_id, parent_observation_id, and an observation type it knows, then keeps the GENERATION records. Each one contributes provided_model_name as the model, input.messages as the messages, output as the answer, and usage_details.input and usage_details.output as usage. tool_definitions ride along with the messages, tool_calls are folded into the output, and session_id becomes the trajectory when it is present.
Cheaper candidates, scored against what shipped
Each kept generation is replayed against cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env. With --evaluator langfuse, your own scorers grade the replays and their pass decisions feed the release gate; without a reachable evaluator the run warns and falls back to the built-in judge, which makes two temperature-zero calls with the reference and candidate positions swapped. Verdicts are confidence lower bounds kept separate per evaluator kind.
A per-step report, then a pull request
The run writes report.md and report.json with a verdict per step family and streams JSONL events as it works. Thin evidence produces an abstention with a named reason. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else; watch reconciles it against CI, and rollback restores the pre-apply files. rightmodeler export --to langfuse pushes trials and verdicts back into a dataset you already review.
Setup
Nothing to install: npx runs the published CLI. Feed it a file of Langfuse observations. The audit runs on the export; your keys are needed for the curl you run yourself, and again if you choose Langfuse as the evaluator.
# export observations via the public API (keys from project settings)
# score the replays with your own Langfuse scorers
# push the trials and verdicts back into a Langfuse dataset
# or let your coding agent drive it
What rightmodeler reads from Langfuse
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: trace_id · parent_observation_id · type
Use cases
Turn Langfuse cost tracking into decisions
Langfuse already tells you what each trace costs. The audit adds the missing column: which steps could run on a cheaper model, with a judged verdict and evidence per step family instead of a hunch about the bill.
Score the audit with the scorers you already run
Pass --evaluator langfuse and your own scorers decide whether a candidate passed, instead of a judge model you did not write. Metric names, rubric versions, and the provider's own pass decision are preserved on each assessment rather than collapsed to a single number.
Build the corpus from a curated dataset
rightmodeler corpus import --from langfuse:dataset builds the replay corpus from a curated dataset instead of raw observations. Curated references are usually human-verified, and reference correctness caps every number downstream, so a verified corpus raises what the audit can prove.
Screen a downgrade before any production trial
Instead of canarying a cheaper model behind a flag and watching dashboards, replay last month's observations and read the per-step verdicts first, judged against outputs your users already accepted.
The honest part
- Only observations whose type is GENERATION become steps. Spans, tools, agents, and chains are read for detection, then dropped with a recorded reason rather than replayed.
- The model has to arrive in provided_model_name. An observation that records it somewhere else is dropped with its reason instead of being guessed at.
- It reads exported files, JSON or JSONL on disk. It does not connect to your Langfuse project, poll the API, or watch new traces arrive; the evaluator, corpus import, and export paths are the only ones that call Langfuse, and only when you ask for them.
- Langfuse cost and score fields are not used as evidence. Candidate calls are priced from what your replay endpoint reports, and marked as an estimate when the provider returns no cost.
- Evidence minimums are per evaluator kind: at least 10 assessed executions, at least two distinct step IDs, and at least five distinct trajectories. Below any of those the family abstains with a named reason instead of recommending.
- It is a report, not a runtime gateway. Langfuse remains your observability layer; rightmodeler never sits in the request path.
Frequently asked questions
How do I export traces from Langfuse?
Call GET /api/public/observations with basic auth (public key as username, secret key as password) and save the data array as JSON, or use a batch export to blob storage in JSONL. Pass the file with --traces; the format is autodetected from the trace_id, parent_observation_id, and type keys.
How do I make Langfuse the evaluator?
Pass --evaluator langfuse with at least one --evaluator-scorer. Credentials come from LANGFUSE_SECRET_KEY and LANGFUSE_PUBLIC_KEY unless you name other variables with --evaluator-api-key-env and --evaluator-public-key-env. Add --evaluator-gate-metric when several scorers are configured, and --evaluator-gate-threshold when the evaluator returns no pass decision of its own.
Can I import a Langfuse dataset instead of traces?
Yes. rightmodeler corpus import --from langfuse:<dataset> builds the replay corpus from a curated dataset and records each case as a curated reference. Since reference correctness caps every downstream number, a human-verified dataset raises the ceiling the audit is measuring against.
Does rightmodeler replace Langfuse?
No. Langfuse observes, scores, and debugs your agent in production; rightmodeler is an offline audit that measures cheaper candidates against the outputs you accepted and reports the evidence. Results can be pushed back with rightmodeler export --to langfuse.
Does my trace data leave my machine?
Replays send step inputs to candidate models through the endpoint you configured, under your own key, the same shape of call your agent already makes. There is no rightmodeler server, and Langfuse is contacted only by the curl you run and by the evaluator, import, or export commands when you choose them.
Will it always find a swap?
No, and that is the point. A family abstains when the evidence is thin, when the excluded fraction runs over the ceiling, or when no candidate clears the quality floor again on held-out cases. A tool that always finds a swap is not measuring anything.
Run the audit on your own traces
The CLI runs from npx, nothing to install; your Langfuse traces do the rest.