Integration · Langfuse
rightmodeler + Langfuse
Langfuse already records what every generation cost and produced. rightmodeler replays those observations through cheaper models and reports, step by step, which swaps hold.
TL;DR
rightmodeler autodetects Langfuse exports by their traceId, observations, and usageDetails keys, then normalizes them with its generic adapter, a best-effort pass that pulls model, input messages, and output per record and keeps the rest in raw. Each successful step is replayed through cheaper OpenRouter candidates and judged against the output you already shipped, with a 0.90 quality floor by default.
How it works with Langfuse
Export observations to a file
Pull observations from GET /api/public/observations (basic auth with your LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY) and save the data array as JSON, or take a JSONL batch export from blob storage. A file or a folder on disk is all the ingester needs.
Autodetected, normalized best-effort
The ingester recognizes Langfuse by its traceId, observations, and usageDetails keys, then hands the records to the generic adapter: model, input messages, and output are extracted per record, and the untouched original rides along in raw. Langfuse's own observation fields (model, input, output, name) are exactly the names it looks for, so flat observation exports land cleanly.
Cheaper candidates, judged against what shipped
Successful single-shot steps are replayed through cheaper OpenRouter models on the exact inputs from your export, and a judge from a different model family (position-swapped, reference-guided) scores each candidate against the output you already shipped, with a 0.90 quality floor by default. Steps with tool calls or loops go to an end-to-end re-run of your pipeline code in a throwaway git worktree instead.
A per-step report, approved by you
The report lays out each proposed swap with its evidence, and the TUI walks you through approve, reject, or hold, per step. Where evidence is weak it abstains, and high-risk families like auth, payments, and migrations are flagged rather than swapped.
Setup
Install the skill once, then feed it a file of Langfuse observations. Nothing connects to your Langfuse project: your keys stay in your own curl call, and the audit runs on the export.
# install the rightmodeler skill
# export observations via the public API (keys from project settings)
# then hand the export to the skill inside Claude Code
What rightmodeler reads from Langfuse
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: traceId · observations · usageDetails
Use cases
Turn Langfuse cost tracking into decisions
Langfuse already tells you what each trace costs. The audit adds the missing column: which steps could run on a cheaper model, with a judged verdict and evidence per step instead of a hunch about the bill.
Right-size without re-instrumenting
No new SDK, no config change, no code in your app. You export observations you already collect, run the skill offline, and read the verdicts. Langfuse keeps observing production; rightmodeler only ever sees the files.
Screen a downgrade before any production trial
Instead of canarying a cheaper model behind a flag and watching dashboards, replay last month's observations and read the per-step verdicts first, judged against outputs your users already accepted.
The honest part
- Langfuse is autodetected, but there is no bespoke adapter: the generic adapter extracts model, input messages, and output per record, best-effort. usageDetails, scores, and nested observation structure stay in the raw record rather than mapped fields.
- It reads exported files, JSON or JSONL on disk. It does not connect to your Langfuse project, poll the API, or watch new traces arrive.
- Steps with tool calls or loops are never judged from the trace alone; they are confirmed by re-running your pipeline end-to-end in a throwaway git worktree with the candidate model injected.
- High-risk step families (auth, payments, migrations, deploys, secrets) are flagged and left on the current model, and weak evidence means the audit abstains rather than recommends.
- It is a report, not a runtime gateway. Langfuse remains your observability layer; rightmodeler never sits in the request path.
Frequently asked questions
How do I export traces from Langfuse?
Call GET /api/public/observations with basic auth (public key as username, secret key as password) and save the data array as JSON, or use a batch export to blob storage in JSONL. Point the skill at the file or folder; the format is autodetected from the traceId, observations, and usageDetails keys.
Why is Langfuse a generic trace source rather than a dedicated adapter?
Detection is bespoke; normalization is not. The generic adapter takes a best-effort pass (model, input messages, output, and name per record) which happens to match Langfuse's own observation fields, and preserves the full record in raw. What it cannot map it does not guess at: steps with tool calls or loops go to end-to-end replay instead of trace-only judging.
Does rightmodeler replace Langfuse?
No. Langfuse observes, scores, and debugs your agent in production; rightmodeler is an offline audit of the traces it produced, answering one question your dashboards don't: which steps can move to a cheaper model without losing quality.
What does it do with Langfuse's cost data?
Not much, today. The generic adapter maps model, messages, and output; Langfuse's cost and usage fields stay in each record's raw copy. Savings in the report are estimated from OpenRouter's blended per-token prices for the incumbent and the candidate, so both sides are priced from the same catalog.
Does my trace data leave my machine?
Replays send step inputs to candidate models through OpenRouter with your own API key, the same shape of call your agent already makes. There is no rightmodeler server, and your Langfuse keys are only ever used by the curl you run yourself.
Will it always find a swap?
No, and that is the point. When evidence is weak or a step is high-risk, the audit abstains and the current model stays. A tool that always finds a swap is not measuring anything.
Run the audit on your own traces
The skill is free on GitHub. One command installs it; your Langfuse traces do the rest.