Comparison · W&B Weave
rightmodeler vs W&B Weave
W&B Weave traces your agent in production, scores it with signals and evaluations you configure, and lets you try other models in its Playground. rightmodeler reads the calls Weave already recorded and settles one narrower question with evidence: which steps can move to a cheaper or newer model?
TL;DR
Complements, working on the same calls. You hire W&B Weave to see and improve an agent in production: as of 2026-09-24, its site describes end-to-end observability of agent sessions, out-of-the-box signals that surface failure modes, a flexible evaluation framework to prevent regressions, guardrails, and a Playground for testing new models against production traces. You hire rightmodeler, a free MIT-licensed CLI, for model right-sizing: its dedicated Weave adapter reads the calls you export, replays the recorded calls through cheaper or newer candidates from your provider's live catalog, judges them against the output you already shipped, and reports agreement, sample size, and abstentions per step family. When a swap clears every gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else, for a human to review and merge. Keep Weave where production is watched; your Weave evaluations are the right last check on any swap.
How they differ
Weave is the platform your agent reports to. rightmodeler runs offline on what Weave recorded.
Can you use both? Yes, and Weave stays put
- Nothing about your Weave setup changes: no new SDK, no re-instrumentation, no change to your weave.op decorators. Weave keeps tracing, scoring, and monitoring production the whole time. rightmodeler reads a file of calls you exported and never connects to your W&B project during a run; your W&B key is used only by the export you run yourself.
- The two answer different questions about the same calls. Weave's evaluations and signals tell you whether the agent clears the bar you set. rightmodeler asks whether each step needs the model it pays for, and a swap it proposes is exactly the change worth running your Weave evaluations against before you merge it.
The supported workflow: Weave calls in, a reviewed pull request out
A dedicated adapter reads Weave call exports. Three commands, run from your repository.
Export the calls with the Weave Python SDK: weave.init on your project, then client.get_calls with the inputs, output, and summary columns, saved as JSON or JSONL. The adapter recognizes Weave rows by their id, trace_id, op_name, and started_at keys, takes the model and messages from inputs, reads output as the answer you shipped, and pulls token counts from the per-model usage in summary. display_name, or op_name when there is none, names the step family, and calls that share a trace_id stay together as one trajectory.
The preview run spends nothing. The full run replays the recorded calls on cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env, judges every replay against the output you accepted, and caps replay and judge spend together at the figure you pass with --max-cost-usd. It writes report.md with a verdict per step family. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers only and cites case-id hashes rather than case content. There is no rightmodeler server, no account, and no telemetry.
# export calls from your Weave project to a file
# preview the pipeline without spending anything
# then run it against your replay endpoint
Which one does the job?
Three situations, with the honest winner named in each.
Overnight, users start complaining that your support agent gives off-topic answers, and you need to find the conversations and the cause.
the right hire: W&B Weave
That is production observability, and it is Weave's job. Per its docs, signals such as the User Frustration and Low Quality Response presets tag agent turns using an LLM judge, a custom signal flags any behavior you describe, and the Agents view lays each conversation out turn by turn; per its site, alerts route to Slack or trigger webhook automations. rightmodeler runs offline on files you exported and has no live view, by design.
Your Weave evaluations are green, yet every op in the agent still calls a frontier model, and nobody can say which ones need it.
the right hire: rightmodeler
Weave's evaluations score the agent against the bar you defined; they do not replay your recorded calls on cheaper models and judge the result against what you shipped. rightmodeler does exactly that per step family, with sample size and abstentions attached, holds the winner to the quality floor on held-out cases, and opens a draft pull request for a multi-step swap only once it is confirmed end to end in a sandbox.
A cheaper model looks promising in the Weave Playground on a handful of production traces.
the right hire: both, together
The Playground shows how models answer the prompts you open, side by side, per Weave's docs. rightmodeler turns that hunch into a per-step verdict across the recorded calls in the family, on your own provider key. If a swap clears, run your Weave evaluation against the draft pull request before you merge it.
The honest part
- Weave watches production continuously. Tracing, preset and custom signals, and alerts tell you what an agent is doing now, and guardrails can block or modify an output before it reaches a user, per its docs. rightmodeler works after the fact on exported files and will never tell you that something broke an hour ago.
- Weave owns the evaluation workflow: versioned datasets, scorers you write or pick from its built-in set, evaluation comparisons, leaderboards, human feedback, and annotation queues. rightmodeler ships none of that. It answers one question, whether each step can move to a cheaper or newer model, and expects your Weave evaluations to remain the bar.
- Weave also covers ground rightmodeler never touches: prompt and model versioning, a Playground for exploring prompts, coding agents that iterate through W&B's skills and MCP server, and hosting on W&B's multi-tenant cloud, a dedicated cloud, or a self-managed instance, per its docs. rightmodeler changes the model a step calls; it does not rewrite prompts, context, or code.
- And rightmodeler's verdicts carry their own limit: they measure how closely a candidate tracked the output you shipped, which is evidence of agreement, not ground-truth correctness. Where the sample is thin, the family abstains with a named reason instead of recommending.
Terms this page uses
- call
- Weave's record of one execution of an op: its inputs, output, timing, and parent, per Weave's docs. A call that recorded a model and plain-text messages under inputs is one step rightmodeler can replay; a call whose conversation carries tool calls, tool definitions, or non-text parts is left out of the replay sample.
- trace
- In Weave, the tree of calls that share one trace_id. rightmodeler keeps those calls together as one trajectory, so a multi-step op chain stays a single unit in the statistics.
- swap candidate
- A cheaper model from your configured provider's live catalog, replayed on one step family's recorded inputs.
- reference evidence
- Agreement between a candidate's output and the output you shipped for that exact call. Evidence of agreement with shipped output, not ground-truth correctness.
- quality floor
- The configurable minimum that the statistical lower bound on a candidate's held-out agreement, adjusted for the number of candidates, must reach before rightmodeler recommends it.
- abstain
- What rightmodeler does when a step family's evidence is too thin: it recommends nothing and names the reason. The current model stays.
Frequently asked questions
Does rightmodeler replace W&B Weave?
No. Weave traces, evaluates, and monitors your agent, and guards its outputs in production. rightmodeler validates cheaper or newer models on calls Weave already recorded, and it answers one question Weave's evaluations do not ask: which steps can move to a cheaper or newer model. A swap it proposes is the change to run through your Weave evaluations before you merge it.
Weave's Playground can already compare models. Why run rightmodeler?
Different evidence. The Playground and the Evaluation Playground compare models on the prompts, traces, or dataset you choose, graded by you or by an LLM judge you configure, per Weave's docs. rightmodeler replays the recorded calls of a step family on cheaper or newer candidates, judges each against the output you shipped, reports sample size and abstentions, and holds the winner to the quality floor on held-out cases before it recommends a swap.
How do I get my calls out of Weave?
With the Weave Python SDK: initialize the client with weave.init("entity/project"), call client.get_calls(columns=["inputs", "output", "summary"]), and write the rows to JSON or JSONL. id, project_id, trace_id, op_name, and started_at come back on every call; those three columns carry the model, the messages, the output, and the token usage the adapter maps. The first command in the workflow above does it in one line.
Does rightmodeler connect to my W&B account?
No. Your W&B key is used by the export you run yourself. Once the calls are on disk rightmodeler runs against them offline, and its model calls use the key for the OpenAI-compatible endpoint you replay through. Recorded inputs and outputs leave your machine only during the run, for replay and judging, to that endpoint, under your key. An external evaluator, if you name one, receives what it scores. End-to-end confirmation on the Vercel Sandbox backend runs cases in your own Vercel project.
Is this a comparison with all of Weights & Biases?
No. This page compares rightmodeler with W&B Weave, the Weights & Biases product for tracing and evaluating agents and LLM applications. The rest of the W&B platform, such as experiment tracking, sweeps, and the model registry, serves model training work and sits outside this comparison.
Will rightmodeler always find a cheaper or newer model?
No, and that is the point. A family abstains when the evidence is thin or the excluded fraction runs over the ceiling, and gets no recommendation when no candidate clears the quality floor on held-out cases. A report that says your current models already earn their cost is a valid result.
Start with the traces you already have.
The CLI runs from npx, nothing to install, and validates cheaper or newer models on your own traces.