Comparison · Braintrust
rightmodeler vs Braintrust
Braintrust grades your agent against a bar you authored: datasets, scorers, rubrics. rightmodeler grades cheaper models against the outputs you accepted, and asks one question Braintrust does not: which model belongs at each step?
TL;DR
Not competitors. Braintrust's job is grading: your agent, held to a bar you built from versioned datasets, scorers, and online scoring of live traffic. rightmodeler's job is one decision that bar was never asked to make: which model belongs at each step. The CLI, on npm and free on GitHub, reruns your real traces on cheaper candidates and scores every candidate against the output you accepted, reporting per step what held up, on how many samples, and where it declined to call a winner. Keep Braintrust. An approved swap then has one more gate to clear: your eval suite.
The words this page turns on
Both tools talk about judges, scores, and references. The difference is authorship: who writes the reference the grading happens against.
- trace
- The step-by-step log your agent already produced: each model call with its inputs, outputs, and tool calls. Braintrust logs these today; the audit reads them from exported files.
- authored bar
- What Braintrust measures against: datasets you curated, scorers you wrote (code-based or LLM-as-a-judge), rubrics you defined. It exists because you built it before the grading began.
- accepted output
- What rightmodeler measures against: the answer your agent actually shipped and you accepted. It already exists in your traces, so there is nothing to build first.
- reference evidence
- Per-step agreement between a cheaper candidate and the accepted output. Evidence of agreement with shipped output, not proof of correctness; the production result is the reference, not ground truth.
- quality floor
- The minimum agreement score a candidate must clear before the audit will recommend a swap. The floor is configurable; below it, no recommendation.
- abstain
- The audit's refusal to call a winner when evidence is thin: too small a sample, or no reliable way to check. Abstentions are reported next to every recommendation.
A model change with Braintrust alone
- Model comparison runs through datasets and scorers you author first; Loop and the playground can test a new model against the bar you maintain.
- The verdict is a score against your criteria, aggregated across the suite.
- Cost shows up in metrics, but nothing asks whether a cheaper model would have matched what you shipped at a given step.
A model change with rightmodeler on top
- The reference set already exists: the outputs your agent produced in the traces you fetched. No dataset to curate, no scorer to write.
- Each step is replayed through cheaper candidates and judged against the accepted output (deterministic checks like tests, builds, schemas, and valid tool calls first; a cross-family judge, meaning one from a different model family than either model being compared, only when nothing stronger applies), guarded by the quality floor.
- You approve every swap yourself, thin evidence produces an abstention instead of a recommendation, and your Braintrust evals confirm the change before it ships.
Which one do you hire?
Three concrete situations, with the honest winner named in each.
You need to know within the hour that tonight's deploy started degrading answer quality in production.
the right hire: Braintrust
Braintrust's online scoring evaluates production traces automatically as they are logged, with quality gates and alerts on drops. rightmodeler is an offline report on files you fetched; it has no live monitoring seat, by design.
Every eval in your suite is green, yet the monthly inference invoice keeps climbing, and the suite has no opinion on which steps are overpaying.
the right hire: rightmodeler
Braintrust scores your agent against the bar you defined; it does not replay individual steps through cheaper candidates and judge them against the output you shipped. rightmodeler answers exactly that question, per step, with reference agreement, sample size, and abstentions attached.
A new frontier model just launched and you want to know what it does to your agent.
the right hire: both, together
Braintrust's Loop and playground can test the new model against the datasets and scorers you maintain, measured against your defined bar. rightmodeler replays it against your shipped outputs step by step and tells you where it earns its price. Different evidence, and both are legitimate.
rightmodeler runs on the traces Braintrust already logs
No re-instrumentation, no new telemetry, no abandoning your stack.
If Braintrust logs your agent today, the input to an audit already exists. You export rows you already have, save the files to disk, and run the CLI on them. The audit never touches your Braintrust account. It runs on the files you already exported, and replays go through your selected replay provider on your own key. Nothing runs on our side: no server, no account, no telemetry. The integration page covers the export and field-mapping details.
Braintrust stays on production duty throughout. The division of labor is clean: rightmodeler proposes the swap with the evidence attached, and validation stays with the eval suite you already trust. The report recommends a model, not a route, and you tip the scale.
The honest part
- Braintrust watches production continuously: scalable trace ingestion, live monitoring, online scoring, and alerts. rightmodeler works after the fact, on exported files, and will never tell you something broke an hour ago, by design.
- Braintrust owns the whole eval lifecycle: versioned datasets, code and LLM scorers, immutable experiments, human review queues, and CI quality gates. rightmodeler ships none of that. It answers one question and expects your eval suite to remain the bar.
- Braintrust is enterprise observability infrastructure: SOC 2 Type II, hybrid deployment, granular permissions, and Brainstore, a database built for querying traces at terabyte scale. rightmodeler's CLI is a free report run locally against exported files, not a place your team's traces live.
- And the audit's verdicts carry their own hedge: they report how closely a candidate tracked the output you shipped, which is not the same thing as being right. Where the evidence runs thin, the audit files an abstention instead of an opinion.
Frequently asked questions
Is the audit a replacement for Braintrust?
No. Braintrust instruments your app, monitors production, and runs the eval lifecycle you built. rightmodeler is a report that answers one question those evals do not ask: which model belongs at each step? When the audit proposes a swap, run it through your Braintrust suite before it ships; that is the point of keeping both.
Can rightmodeler read the traces I already have in Braintrust?
Yes. A shipped integration reads exported Braintrust rows: you fetch them with your own key, save the files, and the audit runs on them offline. The integration page walks through the export and what gets mapped.
Braintrust can test new models too. Why add rightmodeler?
Different reference. Braintrust measures a new model against datasets and scorers you author and maintain; rightmodeler judges candidates against the outputs you accepted, per step, with nothing to build first. If you maintain a strong suite, Braintrust's answer is real. The audit adds per-step evidence of where a cheaper model matches your shipped work, and abstains where that evidence is thin.
Is rightmodeler an observability platform?
Not observability. Not a runtime gateway. It never sits in your request path, needs no instrumentation, and keeps no copy of your traces. It is a report you run on your own files; Braintrust remains where your traces live and where production is watched.
Does the audit's judge replace my Braintrust scorers?
No. Your scorers grade your agent against criteria you authored; that remains the bar. The audit prefers deterministic checks and reaches for an LLM judge only when nothing stronger applies, and its verdicts are always agreement with shipped output, not proof of correctness.
Run the audit on your own traces
The CLI runs from npx, nothing to install, and your own traces settle the question.