How it works
How rightmodeler works
Run it from your terminal, your coding agent, or on autopilot. The same loop everywhere, measured on the traces you already have.
Run it
Pick your surface. Same loop underneath.
From your terminal
One command, nothing to install. It finds the traces Claude Code and Codex already left on disk, or asks for a file, and runs free through shortlist. Add a provider only when you want the replay, and estimate projects that spend first.
# needs Node 24 or newer
The first run
It finds your traces, runs free through shortlist, and asks before anything spends.
From your coding agent
One install gives Claude Code and Codex-class agents the runbook to drive the whole loop: plan preview, staged runs, boundary resumes, and an apply dry run before anything ships.
# for Claude Code and Codex-class agents
In CI, on a schedule
The same command runs unattended. Every event streams as JSON lines, and the exit code is the verdict, so a weekly job can audit, stop at a named boundary, and resume.
exits 0 no recommendation · 1 recommendation · 2 needs input · 3 budget reached
On autopilot, self-hosted
Clone the repo, build it, and run the agent on a long-lived Node 24 host with your own GitHub App and model credentials. It rechecks prices and drift on a schedule; once you give the word, it replays and opens each swap that clears every gate as a draft pull request, evidence attached. The hosted version has a waitlist.
TL;DR
rightmodeler replays your real agent traces through cheaper models, judges each output against what you already shipped, then reports the evidence, sample size, and abstentions for your review.
Where traces come from.
Traces are the logs your AI tools already write: the models called, their inputs, and their outputs. If you use Claude Code or Codex, those logs are already on disk, and init finds them automatically. If your app logs to an observability tool like Langfuse, Braintrust, LangSmith, Helicone, or W&B Weave, export a file and point init at it. You never have to produce anything new.
See every supported sourcein ▸ raw traces
- 01
Detect
Point it at the traces you already emit. rightmodeler autodetects the format across 12 sources and folds every run into one per-step schema, with no new SDK and no re-instrumentation.
reads Claude Code · Codex · LangSmith / LangGraph · OpenAI SDK · Langfuse · Braintrust · Phoenix (OpenInference) · OpenTelemetry GenAI · Vercel AI SDK · Helicone · W&B Weave · Bifrost → 1 per-step schema
- 02
Measure
It replays each step through cheaper candidates on your real inputs and measures every output against what you accepted. Each candidate gets a cost delta, reference-agreement score, and evidence count, and it abstains when the evidence is weak.
scores cost · agreement · evidence count → recommendation + confidence · abstain on thin evidence
- 03
Review
You review the plan, and the CLI ships only the swaps you approve as a pull request with the evidence attached. It then watches CI on that PR, and if you change your mind, one command opens the pull request that restores the exact pre-swap state. Never a live intercept. You decide what to change, and when.
applies approved swaps arrive as a pull request · watch reconciles CI · rollback restores byte-exact
out ▸ signed report
How to read confidence.
A cross-family judge scores both output orders, or the evaluator you name grades the replays: Braintrust, Langfuse, LangSmith, or promptfoo, whose assertions can be deterministic checks such as is-json or regex. Every rate is a statistical lower bound, not a point estimate, and a shortlist winner must clear your quality floor again on held-out cases before it is recommended. Evidence counts show what earned the confidence band, and the evidence type limits how high that band can go. Confidence applies only to the prompt, inputs, and runs evaluated. It measures agreement with what you shipped, not proof of correctness.
Not observability. Not a runtime gateway.
rightmodeler works offline on runs you already shipped. It measures cheaper candidates step by step against the outputs you accepted, and only a swap that clears every gate becomes a draft pull request, which changes model identifiers and nothing else and waits for you to review and merge. Your observability tool keeps recording, any gateway keeps routing live traffic, and rightmodeler never sits in the request path.
FAQ
- What do I need to run it?
- Node 24 or newer, and the traces your AI tools already write. If you use Claude Code or Codex in your project, init finds their traces automatically. A provider key is only needed when you replay, and `npx rightmodeler estimate --base-url <provider>/v1` projects that spend first with zero paid calls.
- Which traces are supported?
- 12 formats, autodetected: Claude Code, Codex, LangSmith / LangGraph, OpenAI SDK, Langfuse, Braintrust, Phoenix (OpenInference), OpenTelemetry GenAI, Vercel AI SDK, Helicone, W&B Weave, and Bifrost. rightmodeler folds them all into one per-step schema, so you point it at the traces you already emit, with no new instrumentation.
- Does it touch production?
- No. rightmodeler replays your past traces offline and produces a report plus a repo edit. It never sits in your request path, routes live traffic, or adds latency. It is not a runtime gateway.
- Do you store my data?
- It runs locally on your own traces and your own replay provider key. Replays call your selected provider, OpenRouter, the Vercel AI Gateway, or a LiteLLM proxy, using your key; there is no rightmodeler server holding your traces.
- Can I use my existing eval framework?
- Yes. Braintrust, Langfuse, LangSmith, and promptfoo can score the replays instead of the built-in judge, and a reachable configured evaluator is always preferred. You can also build the case set from a curated dataset you already maintain, which raises how much the audit can certify, and push trials and verdicts back to your platform when the run completes.
- Can my coding agent run it?
- Yes. One install adds a runbook that Claude Code and Codex-class agents follow to drive the whole loop: plan preview, staged runs with resumable boundaries, and an apply dry run before anything ships. The runbook drives the same CLI through npx.