Comparison · LangSmith
rightmodeler vs LangSmith
LangSmith traces, evaluates, and deploys your agent, and its Engine now turns production traces into diagnosed issues and proposed fix pull requests. rightmodeler reads the runs LangSmith already recorded and settles a narrower question with evidence: can a cheaper or newer model take this step without losing what you accepted?
TL;DR
Different job, same traces. You hire LangSmith to run the agent lifecycle: trace trees, dashboards and alerts, evaluators on production traces, deployment, and, as of 2026-09-22, Engine, which clusters recurring failures into issues, diagnoses each root cause against your traces and connected code, and proposes the fix as a pull request. You hire rightmodeler, a free MIT-licensed CLI, to validate swaps to cheaper or newer models: it replays the LLM runs you export from LangSmith through cheaper or newer candidates from your provider's live catalog, grades each one against the output you already accepted, and reports agreement, sample size, and abstentions per step. When a swap clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else, for a person to review and merge. Not observability. Not a runtime gateway.
What LangSmith Engine does, as of 2026-09-22
- Per LangChain's docs, Engine is the LangSmith agent for agent engineering. It scans your tracing projects on a schedule, clusters recurring failures into prioritized issues, and diagnoses each root cause against your traces and, when you connect a GitHub repository, your source code. From an issue it opens a pull request with the proposed code or prompt fix in that repository, and its changelog for late August 2026 says those pull requests now open automatically. It keeps each issue current by attaching new traces that match the same failure pattern, reopens an issue when the failure comes back, and turns the traces behind an issue into ground-truth dataset examples so you can test the fix offline before it ships.
- That is a full loop from production trace to reviewed fix, and for agent failures it is the right tool. Its scope is failures. Its documented issue categories run from Agent looping and Hallucination to Wrong tool and Tracing quality, and none of them is about which model a step should run on or whether a cheaper one would hold up there. You can tell Engine to prioritize Cost & Tokens, and its Context explosion category flags runaway token counts, but its docs describe no step replayed on other models. Engine's own reasoning runs on LangChain-managed inference, with no bring-your-own-key option, per its docs.
- LangSmith also ships an LLM Gateway, in beta as of 2026-09-22 per its docs: one LangSmith key in front of several providers, with spend limits, rate limits, fallbacks, and redaction applied to live model calls. It governs the model you name. Which model to name at each step is still your call, and rightmodeler gathers evidence for one part of it: whether a cheaper or newer model holds up.
- rightmodeler is narrower on purpose. It does not watch production, cluster failures, or edit prompts or code, and it cannot see a run until you export it. It answers one question per step with receipts, and its pull request changes model identifiers and nothing else.
Side by side
One note on words: reference agreement means how closely a candidate's output matches the one you already shipped for that step.
The situations that decide it
Pick the row that sounds like your week. Each names the honest winner.
The same failed tool call keeps turning up across your production runs, and you need the root cause and a fix before it recurs.
the right hire: LangSmith
This is what Engine is built for, per its docs: it clusters the failing traces into one issue, diagnoses the root cause against your traces and connected code, proposes the fix as a pull request, reopens the issue if the failure comes back, and turns the evidence traces into dataset examples for testing the fix. rightmodeler has no live view and never touches prompts or code.
Your LangGraph agent runs every node on a frontier model. The bill keeps climbing, and nobody can say which nodes need that model.
the right hire: rightmodeler
Every model call lands in the export as its own LLM run, and rightmodeler groups runs into step families by run name, so give each node's model call its own run_name to get a verdict per node. It replays those runs as text on cheaper or newer candidates from your provider's live catalog, grades them against the outputs you accepted, and leaves the frontier model in place wherever no cheaper candidate clears every gate. A run that carries tool calls or tool definitions is left out of the replay sample, so a node that calls tools can end in an abstention. Only a swap that clears every release gate, including the quality floor on held-out cases, becomes a draft pull request.
Your team already runs Engine on production traces and also wants the model bill down without adding risk.
the right hire: both, together
Keep Engine on failures and give rightmodeler the cost question. Export the same project's LLM runs, and if you already grade with LangSmith scorers, pass --evaluator langsmith so those scorers, not a judge you did not write, grade each replay. Engine's fix pull requests and rightmodeler's model-only drafts arrive as separate reviews, so each change is judged on its own evidence.
LangSmith alone
- Trace trees, dashboards and alerts on cost, latency, and errors, plus LLM-as-judge, code, and multi-turn evaluators run on production traces.
- Engine, as of 2026-09-22: recurring issues found on a schedule, root causes traced into your connected code, fixes proposed as pull requests, and dataset examples built from the failing traces.
- Deployment to run agents, and an LLM Gateway, in beta, that applies spend limits, rate limits, and fallbacks to live model calls.
LangSmith with rightmodeler beside it
- Everything on the left stays. rightmodeler reads the LLM runs LangSmith already recorded and adds a per-step answer to one question: does this step need the model it pays for?
- Cheaper candidates come from your configured provider's live catalog and are graded against the output you accepted, either by a judge from a model family neither side belongs to or by your own LangSmith scorers.
- A swap has to clear the quality floor on held-out cases, multi-step swaps are confirmed end to end in a sandbox, and thin evidence ends in an abstention. What clears every release gate becomes a draft pull request that a person reviews and merges.
How a validation run reads your LangSmith exports
No new instrumentation. Your LangSmith key is used to export runs, and by the CLI only if you choose LangSmith scorers or datasets.
Pull the LLM runs from your project with the LangSmith SDK, for example client.list_runs(project_name=..., run_type="llm"), and save them as JSON or JSONL with one run per record. The CLI detects the format from the trace_id, run_type, and dotted_order keys, orders each trace by dotted_order, and takes the model from extra.metadata.ls_model_name. It reads files only: there is no rightmodeler server, no account with us, and no telemetry.
The run writes report.md with a verdict per step family: the candidates tried on your recorded inputs, the agreement each earned against the output you accepted, the sample size behind it, and an abstention with a named reason wherever the evidence is thin. rightmodeler report --output json prints the same report as JSON. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers only and requests review from the owners of the files it touches. It never merges.
# preview the pipeline against the export, without spending
# replay and judge on your own provider key
# or let your own LangSmith scorers grade the replays
The words the report uses
- swap candidate
- A cheaper model shortlisted from your configured provider's live catalog and replayed on one step's recorded inputs.
- reference evidence
- Scores measured against the output you actually shipped for that exact step. Evidence of agreement with shipped output, not ground-truth correctness: the production result is the reference.
- quality floor
- The minimum agreement, at a statistical lower bound, that a candidate must reach on held-out cases before rightmodeler will recommend a swap. Configurable, and anything below it is a no.
- cascade risk
- The chance that a cheaper model at one step quietly degrades the later steps that build on its output. Swaps that could cascade are confirmed end to end in a sandbox, by running your app on recorded cases, before they are recommended.
- abstain
- rightmodeler's decision to recommend nothing for a step when the sample is small or the evidence incomplete, with the reason named. The current model stays.
Frequently asked questions
Doesn't LangSmith Engine already find cheaper models?
Not per its docs as of 2026-09-22. Engine's documented issue categories cover agent failures such as looping, hallucination, wrong tools, and truncated responses, and none of them is about which model a step should run on. You can ask it to prioritize Cost & Tokens, and it flags context explosion, but its docs describe no step replayed on other models and no measurement of whether a cheaper one holds up. That measurement is the core of what rightmodeler does.
Both open pull requests. What is different about rightmodeler's?
Engine's pull request carries a proposed code or prompt fix for a diagnosed failure. rightmodeler apply opens a draft that may only change model identifiers: a diff check refuses anything else, the body carries the evidence table for each step family, and review is requested from the owners of the changed files. It opens only after the swap clears every release gate, and the CLI never merges; a person does.
Can't LangSmith experiments already compare models?
They can, honestly. Build a dataset, define evaluators, and an experiment will compare models across it, and Engine can now seed that dataset from failing traces. rightmodeler starts from the outputs you already accepted, so no dataset is required, and the verdict comes back per step with sample size and abstentions attached. If you keep curated LangSmith datasets and scorers, rightmodeler can use both: rightmodeler corpus import --from langsmith:<dataset> builds the corpus, and --evaluator langsmith lets your scorers grade.
Does rightmodeler connect to my LangSmith account?
Only when you ask it to. rightmodeler reads exported files on your machine, and replays run on your own provider key. The CLI calls the LangSmith API only on the evaluator path (--evaluator langsmith) or for corpus import, using the key in LANGSMITH_API_KEY unless you name another variable.
Is LangSmith open source?
The platform is not. LangSmith, including Engine and the LLM Gateway, is a commercial product that LangChain hosts, and self-hosting it takes a license from LangChain, per its docs. Its client SDK is MIT-licensed, and that SDK is all the export step needs. LangChain's open-source work includes the frameworks: LangChain, LangGraph, and Deep Agents. The rightmodeler CLI is MIT-licensed.
Does every validation run end in recommended swaps?
No, and it is designed not to. Steps with thin evidence get an abstention rather than a swap, and a clean bill is a possible outcome: the report can come back with no swap to make, because no cheaper candidate cleared every gate.
Start with the traces you already have.
The CLI runs from npx, nothing to install, and validates cheaper or newer models on your own traces.