Integration · LangSmith
rightmodeler + LangSmith
Your LangSmith traces already hold the evidence. rightmodeler replays them through cheaper models and proves, step by step, which swaps are safe.
TL;DR
rightmodeler reads exported LangSmith run trees with a dedicated adapter, replays each successful step through cheaper candidates on OpenRouter, and judges every output against the one you already shipped, so a LangSmith-instrumented agent gets right-sized without new instrumentation.
How it works with LangSmith
Export a run tree
Pull runs with client.read_run(run_id, load_child_runs=True) or a bulk export from your LangSmith project, and save the JSON anywhere on disk, as one file or a folder of them.
Autodetected, then normalized
The ingester recognizes LangSmith by its run_type, trace_id, and parent_run_id keys, walks the run tree, and folds every LLM, tool, and chain run into one per-step schema. LangGraph runs arrive the same way: each node becomes a step.
Cheaper candidates, judged against what you shipped
Single-shot steps are replayed through cheaper OpenRouter models on the exact inputs from your trace, and a judge from a different model family scores each candidate against the output you already shipped, with a quality floor of 0.90 by default. Steps with tool calls or loops are confirmed end-to-end by re-running your pipeline code instead.
A per-step report you approve
The report lists every proposed swap with its evidence and confidence, and the TUI walks you through approve, reject, or hold for each step, never all-or-nothing. Where the evidence is weak, it abstains and the current model stays.
Setup
Install once, then point the skill at an exported run tree. Nothing connects to your LangSmith account; the audit runs on files you export.
# install the rightmodeler skill
# inside Claude Code, hand the export to the skill
What rightmodeler reads from LangSmith
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: run_type · trace_id · parent_run_id
Use cases
Right-size a LangGraph agent
Every node in a LangGraph run lands as its own step, so the audit shows which nodes genuinely need a frontier model and which are paying for capability they never use.
Prove a swap before you ship it
Instead of trialing a cheaper model in production and watching dashboards, replay last week's run trees and read the verdict per step, judged against outputs your users already accepted.
Turn traces into a cost review
The report itemizes what each task family costs today and which proven swaps change that, with evidence attached: a model bill your team can actually act on.
The honest part
- It reads exported run trees as JSON or JSONL files. It does not connect to your LangSmith account, poll projects, or watch new runs.
- Steps with tool calls or loops are not judged from the trace alone; they are confirmed by re-running your pipeline end-to-end with the candidate model injected.
- High-risk steps like auth, payments, and migrations are flagged and left on the current model, and weak evidence means abstain rather than a recommendation.
- It is a report, not a runtime gateway. LangSmith keeps observing your agent; rightmodeler never sits in the request path.
Frequently asked questions
How do I get my traces out of LangSmith?
Use the Python SDK, where client.read_run(run_id, load_child_runs=True) returns the full run tree, or run a bulk export from your project, and save the result as JSON or JSONL. Point the skill at the file or folder; the format is autodetected.
Does rightmodeler replace LangSmith?
No. LangSmith observes your agent in production; rightmodeler is an offline audit that answers one question your dashboards don't: which steps can move to a cheaper model without losing quality. They work on the same traces.
Does it work with LangGraph?
Yes. LangGraph runs are recorded as LangSmith run trees, and each node becomes a step in the audit. For graph steps that call tools or loop, the skill re-runs your pipeline end-to-end, and it can even find the entrypoint from langgraph.json, so cascade effects surface before you swap.
Which candidate models does it try?
Cheaper models from OpenRouter's catalog that support what the step actually needs (tool calling, structured output, enough context) and are priced strictly below the current model. Replays run with your own OpenRouter key, and a judge from a different model family scores each output.
Does my trace data leave my machine?
Replays send step inputs to candidate models through OpenRouter with your own API key, the same shape of call your agent already makes to a provider. There is no rightmodeler server, and nothing else holds your traces.
Will it always find savings?
No, and that's deliberate. When the evidence is weak or a step is high-risk, the audit abstains and the current model stays. A tool that always finds a swap is not measuring anything.
Run the audit on your own traces
The skill is free on GitHub. One command installs it; your LangSmith traces do the rest.