Integration · W&B Weave
rightmodeler + W&B Weave
Weave already records every op call your agent made, with its inputs, its output, and its token summary. rightmodeler reads those calls with a dedicated adapter, replays the recorded model calls against cheaper or newer candidates, and reports the evidence per step family.
TL;DR
rightmodeler has a dedicated Weave adapter. It recognizes call rows by their id, trace_id, op_name, and started_at keys, takes the model and messages from inputs, reads the output field as the answer you shipped, and pulls token counts out of the per-model usage map in summary. Calls sharing a trace_id become one trajectory, so a multi-step op chain stays a single unit in the statistics.
How it works with W&B Weave
Export calls from your project
Initialize the client against your project with weave.init("entity/project"), then read the rows with client.get_calls(columns=["inputs", "output", "summary"]) and save them as JSON or JSONL on disk. id, project_id, trace_id, op_name, and started_at are always included, and those three columns carry everything else the adapter maps.
Autodetected, then mapped field by field
The adapter recognizes the format from id, trace_id, op_name, started_at, and an inputs object. It reads the model from inputs.model, takes messages from inputs.messages, appends inputs.tools whenever that list is non-empty, and reads token counts from summary.usage keyed by that model. display_name, or op_name when there is none, names the step family.
Cheaper or newer candidates, scored against your output
Each recorded call with a plain-text conversation and an empty or missing inputs.tools list is replayed against cheaper or newer candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env, so OpenRouter, the Vercel AI Gateway, or a LiteLLM proxy all work unchanged. A configured external evaluator scores the outputs when it is reachable; otherwise the built-in judge runs two temperature-zero calls with the reference and candidate positions swapped. Verdicts rest on statistical lower bounds kept separate per evaluator kind.
A report, then a pull request you review
The run writes report.md with a verdict per step family, rightmodeler report --output json prints the full JSON report from the local store, and --output jsonl streams events as the run works. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else and cites case-id hashes rather than case content. watch reconciles that pull request against CI and review comments, and rollback opens another pull request that restores the pre-apply files, refusing unless every restored digest matches what was recorded.
Setup
Nothing to install: npx runs the published CLI. Point it at your exported calls. Nothing connects to Weights & Biases during a validation run: your key is used by the export you run yourself.
# export calls from your Weave project to a file
# preview the pipeline without spending anything
# then run it against your replay endpoint
# or let your coding agent drive it
What rightmodeler reads from W&B Weave
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: id · trace_id · op_name · started_at
Use cases
Price the ops your evals already cover
Weave tells you how your ops score against the evaluations you wrote. rightmodeler answers the adjacent question: which of those ops are paying frontier prices for a result a cheaper model reproduces, with a verdict per step family.
Right-size a traced op chain
Because every call carries op_name, related calls group into step families and the report reads per family rather than per request. A chain where one op genuinely needs the big model and three do not shows up as exactly that.
Reuse traces you already collect
No new SDK, no re-instrumentation, no change to your decorators. You export calls Weave already stores, run rightmodeler offline, and read the verdicts; Weave keeps tracing production throughout.
The honest part
- It reads exported calls on disk, JSON or JSONL. It does not connect to your Weave project, poll its API, or watch new calls arrive.
- Only calls that recorded a model and messages under inputs can be replayed. Ops that wrap non-model work are read and then dropped with a recorded reason rather than counted as steps.
- A call with a non-empty inputs.tools list, or whose recorded conversation carries tool calls or non-text parts, is read but not replayed yet: replay sends recorded text conversations, so those cases are left out of the replay sample, and the recorded_messages_not_replayable warning says how many.
- Token usage comes from the summary.usage entry for the call's own model. Calls whose summary carries no usage for that model fall back to the first usage entry present, and a call with none records zero.
- Evidence minimums are per evaluator kind: at least 10 assessed executions, at least two distinct step IDs, and at least five distinct trajectories. Below any of those the family abstains with a named reason instead of recommending.
- Steps whose output can feed later model-authored steps are confirmed by running your pipeline in a container with the candidate swapped in. That needs a --modeb-config file naming the image and the step map; without it the recommendation is reported as unconfirmed.
- It is not a runtime layer. Weave keeps tracing and evaluating your agent; rightmodeler never sits in the request path.
Frequently asked questions
How do I export calls from Weave?
Initialize the client with weave.init("entity/project"), call client.get_calls(columns=["inputs", "output", "summary"]), and write the rows to JSON or JSONL. id, project_id, trace_id, op_name, and started_at come back on every call; those three columns carry the model, the messages, the output, and the token usage the adapter maps.
Does rightmodeler replace my Weave evaluations?
No. Weave evaluations measure whether your agent clears a bar you defined. rightmodeler measures whether a cheaper or newer model matches the output you already shipped, step family by step family. A proposed swap is exactly the change worth running your Weave evaluations against before it ships.
Do I need a Weights & Biases account to run rightmodeler?
Only to produce the export. Once the calls are on disk rightmodeler runs against them offline, and the only credential it needs is the key for whichever OpenAI-compatible endpoint you replay through.
How does it group my calls?
By trace_id. Calls sharing a trace become one trajectory, which is the clustering unit for the statistics: repeated cases from a trajectory are resampled at the trajectory level rather than treated as independent trials.
Does my call data leave my machine?
Only during the run: recorded inputs go to candidate models, and recorded responses go to the judge, through the endpoint you configured, under your own key. An external evaluator, if you name one, receives what it scores. There is no rightmodeler server and no account; the export file and the report stay on your disk.
Will it always find a cheaper or newer model?
No, and that is the point. A family abstains when the evidence is thin or the excluded fraction runs over the ceiling, and nothing is proposed unless a candidate clears the quality floor on held-out cases. A tool that always finds a swap is not measuring anything.
Start with the traces you already have.
The CLI runs from npx, nothing to install, and validates cheaper or newer models on your traces from W&B Weave.