Coming soon · by rightmodeler

The last model migration you do by hand.

A new model ships. rightmodeler agent replays it against your real traces, prices the swap, and opens a pull request with the evidence attached. You review it like any other change.

Get early access. One note when it opens, no spam.

How it decides

rightmodeler agent · pull requestillustrative

swap: summarize step to gpt-5.4-mini

quality0.94 vs 0.95 shipped · floor 0.90 · medium
p95 latency-38% vs current
list price$5.00 → $0.75 in · $30.00 → $4.50 out · /1M

steps/summarize.ts

- model: "gpt-5.6",

+ model: "gpt-5.4-mini",

cheaper per token · gpt-5.6 to gpt-5.4-mini · list prices

Passed:replay: 214 tracesPassed:judge: passPassed:cascade: clear

The deliverable

A one-line diff with the receipts attached.

TL;DR

rightmodeler agent watches every model release, proves the ones that beat your current stack on your own traces, and ships the swap as a pull request in your repo. Model migrations become code review.

The loop

Watch, replay, judge, open the PR. Continuous, and always inside your guardrails.

Watch

Every release, every provider, tracked live. Candidates get flagged per step the moment they ship.

Replay

Candidates rerun your real traces end to end in a sandboxed worktree, so cascade failures surface early.

Judge

Each output is judged against what you already shipped, cross-family, with your quality floor as the bar.

Open the PR

Diff, evidence, and confidence, opened for your review. Weak evidence means no PR.

Your preferences are the policy.

The agent moves only inside guardrails you set. A config in your repo decides what counts as better, what it may touch, and what it must never go near.

quality_floor0.90 · judged against shipped outputs
min_saving20% per step
latencyp95 within current budget
providersallow openai · anthropic · google · meta
never_touchauth_code_edit · payments_*
mergeopen PR only · never auto-merge

illustrative config

Runs where your work lives.

GitHub ActionsScheduled CIYour API keys

Nothing sits in your request path. The agent wakes in CI, does its work, opens a PR, and goes back to sleep.

  • Receipts on every PR

    The diff ships with quality scores, cost deltas, latency, and the replayed traces behind them. When a reviewer asks why, the answer is already attached.

  • It abstains

    No candidate clears your floor, no PR. A tool that always finds a swap is not measuring anything.

The proof engine behind the agent is the rightmodeler skill, and you can run it on your own traces today.

View on GitHub
RelatedHow the proof worksCrucible, the analytics suiteRead the vision: the Tuesday problem

FAQ

What is rightmodeler agent?
An autonomous agent that keeps every step of your AI stack on the right model. It watches new model releases, replays them against your real traces, judges each output against what you already shipped, and opens a pull request in your repo when a swap clears your quality floor and preferences.
When can I use it?
rightmodeler agent is in active development. Join the waitlist and we will send one note when early access opens. The proof engine behind it, the rightmodeler skill, is available now on GitHub.
Does it merge changes on its own?
No. The agent opens pull requests; merging stays with you. A preferences file in your repo sets the guardrails: quality floor, minimum saving, latency budget, provider allowlist, and steps it must never touch.
What does it evaluate against?
Your own traces and the outputs you already shipped, not public benchmarks. Judging is cross-family, position-swapped, and reference-guided, and the agent abstains instead of opening a PR when the evidence is weak.
How is it different from the rightmodeler skill?
Same proof loop, different cadence. The skill is an audit you run when you want it. The agent runs that loop continuously in CI, watches every release, and turns the result into a pull request you review.