Manifesto
Measure it. Don't guess.
A model downgrade is a real decision. It deserves evidence, not a vibe.
The default is over-provisioned.
The biggest model on every step feels safe. Then the bill compounds, call by call, and nobody can say which step earned it.
model usage · Juneillustrative
route · gpt-5.6$1,180
extract · gpt-5.6$940
summarize · gpt-5.6$2,310
judge · gpt-5.6$860
rerank · gpt-5.6$1,040
embed · gpt-5.6$620
month to date$5,290 and climbing
Evidence beats vibes.
Leaderboards are not your workload. Every candidate is replayed on your own traces and judged against what you shipped.
A category, not a feature.
Evidence-backed model downgrading: detect, measure, review. A report you run today, pull requests next, continuous with Crucible.
Review the evidence
Illustrativerightmodeler-agent
swap: summarize step to gpt-5.4-mini
Q 0.94 · 85% cheaper · 214 traces replayed
Replays, scores, and confidence attached. The merge stays yours.
CloseMerge
It can say no.
Weak evidence means abstain.
Your traces are the benchmark.
Judged against what you shipped, never a leaderboard.
Nothing swaps on its own.
Risks flagged, evidence attached, merge yours.
FAQ
- What is evidence-backed model downgrading?
- Reviewing a cheaper candidate only after measuring its output against the output you accepted for the same input. The decision is backed by replays, reference-agreement scores, sample size, and confidence, not a benchmark or a hunch.
- How is this different from observability?
- Observability shows you what happened. It doesn't replay your steps through cheaper models, measure agreement with accepted outputs, or prepare a repo edit. rightmodeler reports the evidence and applies only the changes you approve.
- Is it safe to downgrade automatically?
- rightmodeler never swaps on its own. It abstains when the evidence is weak, flags cascade risk, and leaves the final call, and the repo edit, to you.