Integration · LangSmith
rightmodeler + LangSmith
Your LangSmith traces already hold the inputs and accepted outputs. rightmodeler replays them through cheaper models and reports candidate agreement, evidence, sample size, and abstentions step by step, scored by LangSmith itself when you point it there.
TL;DR
rightmodeler has a dedicated LangSmith adapter. It recognizes exported runs by their trace_id, run_type, and dotted_order keys, maps the runs whose run_type is llm, and takes the model from extra.metadata.ls_model_name with messages from inputs.messages and the answer from outputs. LangSmith can also be the evaluator: pass --evaluator langsmith with a dataset and scorer names, and the built-in judge is used only when nothing better is reachable.
How it works with LangSmith
Export a run tree
Pull runs with client.read_run(run_id, load_child_runs=True) or a bulk export from your LangSmith project, and save the JSON anywhere on disk, as one file or a folder of them.
Autodetected, then mapped field by field
The adapter recognizes LangSmith by trace_id, run_type, and dotted_order, then keeps the runs whose run_type is llm. Each one contributes its model from extra.metadata.ls_model_name (or extra.invocation_params.model), its messages from inputs.messages, its answer from outputs, and its usage from the run's prompt_tokens and completion_tokens. dotted_order orders the steps, and thread_id, session_id, or conversation_id becomes the trajectory when present.
Cheaper candidates, judged against what you shipped
Each kept run is replayed against cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env. With --evaluator langsmith, your own scorers grade the replays against the named dataset and their pass decisions feed the release gate; without a reachable evaluator the run warns and falls back to the built-in judge, two temperature-zero calls with the reference and candidate positions swapped. Verdicts are confidence lower bounds kept separate per evaluator kind.
A per-step report, then a pull request
The run writes report.md and report.json with a verdict per step family and streams JSONL events as it works. Thin evidence produces an abstention with a named reason rather than a recommendation. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else, watch reconciles it against CI and review comments, and rollback restores the pre-apply files, refusing unless every restored digest matches.
Setup
Nothing to install: npx runs the published CLI. Point it at an exported run tree. The audit reads the file offline; your LangSmith key is needed only if you choose LangSmith as the evaluator or import one of its datasets.
# preview the pipeline against the export, without spending
# score the replays with your own LangSmith scorers
# or build the corpus from a curated LangSmith dataset
# or let your coding agent drive it
What rightmodeler reads from LangSmith
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: trace_id · run_type · dotted_order
Use cases
Right-size a LangGraph agent
LangGraph runs are recorded as LangSmith runs, so each node that calls a model lands as its own step and groups into a step family by name. The audit shows which families genuinely need a frontier model and which are paying for capability they never use.
Score the audit with the evaluators you already wrote
Pass --evaluator langsmith with the dataset and scorer names you already use, and those scorers decide whether a candidate passed rather than a judge model you did not write. Metric names, rubric versions, and the provider's own pass decision are kept on each assessment.
Build the corpus from a curated dataset
rightmodeler corpus import --from langsmith:dataset builds the replay corpus from a curated dataset instead of raw runs. Curated references are usually human-verified, and reference correctness caps every number downstream, so a verified corpus raises what the audit can prove.
Measure a candidate before you change a model
Instead of trialing a cheaper model in production and watching dashboards, replay last week's run trees and read the verdict per step family, judged against outputs your users already accepted.
The honest part
- Only runs whose run_type is llm become steps. Chain, tool, and retriever runs are read for detection, then dropped with a recorded reason rather than replayed.
- An llm run has to carry both inputs and outputs, and a model in extra.metadata.ls_model_name or extra.invocation_params.model. A run missing either is dropped with its reason instead of guessed at.
- It reads exported run trees as JSON or JSONL files. It does not connect to your LangSmith account, poll projects, or watch new runs; the evaluator and corpus-import paths are the only ones that call LangSmith, and only when you ask for them.
- Evidence minimums are per evaluator kind: at least 10 assessed executions, at least two distinct step IDs, and at least five distinct trajectories. Below any of those the family abstains with a named reason instead of recommending.
- Selection is split-corpus. Candidates are shortlisted on one half of the cases, and the winner has to clear the configured quality floor again on held-out cases against a multiplicity-corrected lower bound.
- It is a report, not a runtime gateway. LangSmith keeps observing your agent; rightmodeler never sits in the request path.
Frequently asked questions
How do I get my traces out of LangSmith?
Use the Python SDK, where client.read_run(run_id, load_child_runs=True) returns the full run tree, or run a bulk export from your project, and save the result as JSON or JSONL. Pass the file or folder with --traces; the format is autodetected from the trace_id, run_type, and dotted_order keys.
How do I make LangSmith the evaluator?
Pass --evaluator langsmith with --evaluator-project-id naming the dataset and at least one --evaluator-scorer. The key comes from LANGSMITH_API_KEY unless --evaluator-api-key-env names another variable. Add --evaluator-gate-metric when several scorers are configured, and --evaluator-gate-threshold when the evaluator returns no pass decision of its own.
Does it work with LangGraph?
Yes. LangGraph runs are recorded as LangSmith runs, and every node that calls a model becomes a step grouped by its run name. Steps whose output can feed later model-authored steps are confirmed by running your pipeline in a container with the candidate swapped in, so cascade effects surface before you swap.
Which candidate models does it try?
Models from the live catalog of whichever OpenAI-compatible endpoint you pass with --base-url: cheaper than the incumbent, and capable of what the step needs. Zero-priced models are left out unless you pass --include-free, and the built-in judge is chosen from a family that is neither the candidate's nor the reference's.
Does my trace data leave my machine?
Replays send step inputs to candidate models through the endpoint you configured, under your own key, the same shape of call your agent already makes. There is no rightmodeler server, and LangSmith is contacted only by the evaluator or corpus-import commands when you choose them.
Will it always find savings?
No, and that is deliberate. A family abstains when the evidence is thin, when the excluded fraction runs over the ceiling, or when no candidate clears the quality floor again on held-out cases. A tool that always finds a swap is not measuring anything.
Run the audit on your own traces
The CLI runs from npx, nothing to install; your LangSmith traces do the rest.