Source: https://www.rightmodeler.com/vs/langsmith # rightmodeler vs LangSmith LangSmith watches, scores, and increasingly runs your agent. rightmodeler reads the run trees it already produced and answers the one question its dashboards leave open: which model belongs at each step. Different job, same traces. You hire LangSmith to watch your agent: step-by-step trace trees, dashboards and alerts, evaluator scores on live traffic, and increasingly to run the agent itself. You hire rightmodeler for one decision the platform does not make for you: which model belongs at each step. The rightmodeler CLI, free on npm and GitHub, replays the run trees you export from LangSmith through cheaper models, judges each candidate against the output you already accepted, and reports the evidence, sample size, and abstentions (steps where the evidence was too thin to call). Not observability. Not a runtime gateway. ## The situations that decide it Pick the row that sounds like your week. Each names the honest winner. ### Your agent failed in production overnight. You need the exact call that broke, what it cost, and why latency spiked. Use: theirs This is live observability, and LangSmith is built for exactly it: walk the trace tree step by step, read cost and latency on every call, and set an alert so the next failure pages you. rightmodeler has no live view at all; it cannot see a run until you export it. ### Your LangGraph agent runs every node on the frontier model. The bill hurts, and nobody can say which nodes actually need it. Use: ours Because each LangGraph node arrives as its own step in the export, the verdict comes back per node, judged against outputs you already accepted: keep the frontier model where the evidence demands it, swap where it doesn't. A spend dashboard shows what each node costs, not which swap would survive. ### Your team is happily instrumented on LangSmith and wants the bill down without ripping anything out. Use: both Keep LangSmith observing. Export the run trees it already holds, run the audit, and approve the swaps that clear your quality floor (the minimum agreement score a candidate must hit before a swap is recommended). Then watch LangSmith's own cost dashboards confirm the new mix in production. ## Side by side One note on words: reference agreement means how closely a candidate's output matches the one you already shipped. | | LangSmith | rightmodeler | | --- | --- | --- | | the seat | a platform you work inside, in and around your production path | a report you run offline; never in your request path | | evidence | evaluator scores over live traffic and datasets you curate | reference agreement, evidence, sample size, and abstentions | | unit of verdict | the run and the experiment | the step; per node on LangGraph runs | | instrumentation cost | instrument once, then every run streams in | zero; exported run trees are the whole input | _langsmith vs rightmodeler_ **LangSmith alone** - Trace trees for step-by-step debugging, dashboards and alerts, and online evaluations scoring live traffic as it happens. - Offline experiments against datasets you curate, pairwise comparisons, and annotation queues for structured human review. - And increasingly a runtime: LangSmith deploys and runs agents. **LangSmith with rightmodeler on top** - Everything on the left stays. The audit reads the run trees LangSmith already holds and adds the verdict the platform does not issue: which model belongs at each step. - Cheaper candidates are judged against the output you accepted, by a judge from a different model family than either candidate, so the expensive model gets no home-field advantage. - You approve every swap. The report recommends a model, not a route, and LangSmith's own dashboards verify the new mix in production. ## How the audit runs on your exports No new instrumentation and no account access. The traces LangSmith already collected are the whole input. Export run trees from your LangSmith project and save them anywhere on disk. The CLI autodetects the format, walks each tree, and reads files only: it never connects to your LangSmith account, and there is no rightmodeler server, no account with us, no telemetry. What comes back is a per-step report: for each step, the cheaper candidates tried on your recorded inputs, the agreement each one earned against the output you accepted, and the sample size behind that score. Steps that call tools or loop are never judged from one replay; they are confirmed end to end in a throwaway sandbox so cascading failures surface before you swap. A candidate only becomes a recommendation when it clears the configurable floor. And when the sample is too small to call, the report records an abstention instead of a guess, so nothing sensitive gets swapped on thin evidence. ```bash # # export run trees from your project, then hand the folder to the CLI claude "run the rightmodeler skill on ./exports/langsmith/" ``` ## The words the report uses - **model downgrade audit**: The report the CLI produces: every step of a recorded run, replayed through cheaper candidates, with a per-step recommendation or an abstention. - **reference evidence**: Scores measured against the output you actually shipped for that exact step. Evidence of agreement with shipped output, not proof of correctness: the production result is the reference, not ground truth. - **quality floor**: The minimum agreement score a candidate must clear before the audit will recommend a swap. Configurable, and anything below it is a no. - **abstain**: The audit's decision to make no recommendation when the sample is small or the evidence is mixed. The current model stays. ## FAQ ### Do I still need LangSmith if I run the audit? Yes. LangSmith is the seat you watch and increasingly run your agent from; rightmodeler is a report you run on the run trees you export from it. Nothing gets ripped out: the audit's whole input is data LangSmith already collected, and its output is a model recommendation you carry back to your own repo. ### Can't LangSmith experiments already compare models? They can, honestly. Build a dataset, define evaluators, and an experiment will A/B models across it. rightmodeler starts one step earlier: the reference is the output you already accepted, so there is no dataset to curate and no evaluator to write, and the verdict comes back per step with sample size and abstentions attached. ### How is this different from a runtime gateway routing model calls? A runtime gateway sits in your request path, proxying and metering live model calls. The audit never sits in the request path: it recommends a model, not a route, backs the recommendation with per-step evidence, and you approve every swap before anything changes. ### Does rightmodeler connect to my LangSmith account? No. It reads the files you export, and the audit runs on your machine with your own provider keys. Your instrumentation never notices the audit ran. ### Does every audit end in recommended swaps? No, and it is designed not to. Steps with thin evidence get an abstention rather than a swap, and a clean bill is a possible outcome: the report can come back saying your current model assignments already earn their cost.