Comparison · OpenRouter

rightmodeler vs OpenRouter

One is a gateway that carries your live model calls and now ships its own answer to which model, ranked by what the market spends. The other validates cheaper or newer models offline, judging candidates per step against the outputs you already accepted, and can run its replays on your own OpenRouter key.

Complement · rightmodeler runs on topVisit OpenRouter  (opens in a new tab)

TL;DR

Hire OpenRouter to carry live traffic: one key, one OpenAI-compatible API, a live catalog of 400+ models from 70+ providers, with provider pricing passed through at no markup on inference. Hire rightmodeler to validate whether a cheaper or newer model can take each step of your agent: it replays your recorded traces (a trace is the record of what your agent actually did) through cheaper or newer candidates and judges them against the outputs you accepted. Mostly they compose: rightmodeler can run its replays through OpenRouter, on your own key. They compete on exactly one question, which model, and they answer it from different evidence: Auto ranks candidates by what the whole community spent on that task type over a trailing 7-day window, their Ori Eval grades test cases it generates, and rightmodeler grades against what you shipped.

OpenRouter decides at request time

  • A gateway in your request path. Your app sends a request, OpenRouter forwards it to a model, and the answer comes back. It is in the room every time your agent speaks.
  • When you let it choose, Auto classifies the prompt into one of roughly 30 task types and picks from what the whole community spent on that task type over the last seven days, inside the cost tier you set. The evidence behind the pick is market behavior, never your outputs.

rightmodeler validates before review

  • A validation run, offline, on traces you already have. Not observability. Not a runtime gateway. It replays recorded steps through cheaper or newer candidates from the live catalog.
  • Every candidate is graded against the output you accepted, by a judge drawn from a neutral third model family or by your own evaluator. The built-in judge reads each pair twice with the order reversed, and a case passes only when both readings call it equivalent. The verdict comes with reference agreement (how closely the candidate matched what you shipped: evidence of agreement, not ground-truth correctness), sample size, and abstentions (an abstention is a refusal to recommend when the evidence is too thin). A person merges every swap.

Your stack with the gateway alone

  • You pick the model, or Auto picks one from market behavior. Either way the choice is carried out instantly and graded against your outputs never.
  • Quality is whatever you notice after the switch; the ranking has no memory of your accepted outputs.
  • Changing the model in production means pointing live customer traffic at it; the gateway ships no bench built from your own steps.

The same stack with rightmodeler on top

  • Keep OpenRouter exactly where it is. A validation run works offline, on your key, and never sits in your request path.
  • Candidates are graded step by step against answers you already accepted. The production result is the reference, not ground truth.
  • A quality floor (a configurable minimum agreement score below which no swap is recommended) guards every verdict, and thin evidence produces an abstention instead of a recommendation.

Which one does the job?

Three jobs, and which tool actually gets hired for each.

You are building a new product, have no production traffic yet, and need to pick a starting model for a new feature.

the right hire: OpenRouter

With no traces there is nothing for rightmodeler to replay: no traces, no validation. Ori Eval scans your codebase for model call sites, interviews you, generates eval files with assertions and an LLM judge, and recommends a model. Before real usage exists, that is the right tool.

A cheaper model looks plausible for your extraction step, and you need to know it actually matches what you shipped before you commit.

the right hire: rightmodeler

rightmodeler replays the step's real recorded inputs, and a neutral third-family judge scores each candidate against the output you accepted, with sample size reported and abstentions when the evidence is thin. A 7-day spend ranking has no memory of what you shipped, and a single hand-run trial is one draw from a distribution research shows to be wide and lopsided.

rightmodeler validated a swap you plan to merge, and you also want multi-provider uptime for the model you are moving to.

the right hire: both, together

rightmodeler names a model; where you serve it is up to you. It replayed the candidates through OpenRouter on your key, and serving the new model through OpenRouter afterward buys the one-key breadth and multi-provider uptime rightmodeler never provides.

Use OpenRouter when

  • You want one key and one API in front of every major model family.
  • You want hands-off per-request model choice inside a cost tier.
  • You have no traces yet and want Ori Eval to bootstrap a benchmark.

Use rightmodeler when

  • You want evidence a cheaper or newer model holds up on your steps before you switch.
  • You want per-step verdicts with reference agreement, sample size, and abstentions.
  • Your serving path is fixed (Bedrock, Azure, a company proxy) and the question is the model, not the route.

The same question, different evidence

Both now answer the which-model question. Here is what each answer is made of.

openrouter vs rightmodeler
decides
OpenRouter · per request, at the gateway
rightmodeler · per step, offline, in evidence and a draft pull request you review
reference
OpenRouter · community spend per task type, trailing 7 days
rightmodeler · the output you accepted for that exact step
test cases
OpenRouter · generated by Ori Eval from your codebase and an interview
rightmodeler · your recorded traces, replayed on real inputs
in your request path
OpenRouter · yes, it is the gateway
rightmodeler · never; it runs offline
quality bar
OpenRouter · cost tiers and market rankings
rightmodeler · a configurable quality floor, a cross-family judge, abstentions
who applies the change
OpenRouter · Auto, per request, when you let it choose
rightmodeler · a person, by merging the draft pull request rightmodeler opens

rightmodeler replays through OpenRouter

Setup is a single environment variable.

rightmodeler shortlists cheaper candidates from OpenRouter's live catalog, replays your recorded steps, runs a judge from a third model family, and prices every call at what OpenRouter reports it charged. Setup details live on the integration page.

Your production traffic never passes through the replay key; before and after, requests flow wherever they already flow. If you merge a swap, serve the model there or anywhere else: rightmodeler borrowed OpenRouter for its replays; nothing in what you ship depends on it.

# one key is the entire setup

Setup guide: rightmodeler + OpenRouter

The honest part

  • OpenRouter works on day one with zero history. One key reaches 400+ models, and it keeps production flowing while rightmodeler never touches live traffic at all. rightmodeler needs recorded traces before it can say anything; no traces, no validation.
  • OpenRouter also gives you continuous gateway-side visibility with no pipeline changes: Classifiers (in beta) tag each generation against your taxonomy, and the Activity Explorer shows which models handle which tasks and where spend goes. The rightmodeler CLI validates at a point in time, when you or your CI run it, and never watches live traffic.
  • One more honest number, and it cuts both ways. In the study How Do AI Agents Spend Your Money? (Bai et al., arXiv:2604.22750), a team from Michigan, Stanford, MIT, Google DeepMind, and All Hands AI repeated 500 real GitHub issues four times over on eight frontier models and recorded every token; total usage swung as much as thirtyfold between runs where nothing changed except chance. That variance is why any single-run impression of a model says very little, whichever tool produced it. It is also why rightmodeler reports sample size beside every recommendation and abstains when the evidence is thin. Some validation runs end in abstentions instead of savings, and that possibility is precisely what makes the savings believable when they do appear.

Terms this page leans on

trace
The record of what your agent actually did: each step's inputs, outputs, model, and tool calls, exported from the stack you already run.
reference evidence
Scoring a candidate against the output you accepted in production. This measures agreement with shipped output, not ground-truth correctness.
quality floor
The configurable minimum agreement score a candidate must clear before rightmodeler will recommend a swap.
swap candidate
A cheaper model rightmodeler shortlists, replays, and judges for one specific step.
abstain
rightmodeler's decision to make no recommendation when the evidence or sample is too weak to support one.

Frequently asked questions

Does rightmodeler replace OpenRouter?

No. OpenRouter serves live traffic; rightmodeler validates, offline, on traffic you already served. It is not a runtime gateway; it never sits in your request path. Its replays can even run through OpenRouter, on your key.

OpenRouter's Auto router already picks a model for me. Why validate?

Auto picks per request from what the community spent on that task type over the last week, inside your cost tier. That is a sensible default for a hands-off chat product, and it never looks at your outputs. rightmodeler validates one pinned model per step and shows its evidence: reference agreement with what you shipped, sample size, and abstentions. A model decision deserves evidence, not a vibe.

How is rightmodeler different from Ori Eval?

Ori Eval scans your codebase, interviews you, generates eval files with assertions and an LLM judge, and recommends a model; its test cases come from what it finds in your repo and your interview answers, and it can fold in data files it discovers there, but it does not replay recorded production traces against outputs you accepted. rightmodeler replays your real traces and judges candidates against outputs you already accepted. Neither result is ground truth; rightmodeler measures agreement with what you shipped, which is exactly the bar a swap to a cheaper or newer model has to clear.

Does an approved swap lock me into OpenRouter?

No. You approved a model, not a serving path; ship it through OpenRouter, another gateway, or the provider directly.

What does each one cost?

The rightmodeler CLI is free, on npm and GitHub; you pay for its replay and judge calls on your OpenRouter key, and the report states that spend. OpenRouter's side of the bill is the provider's own pricing with no markup on inference; buying credits carries a small purchase fee.

Start with the traces you already have.

The CLI runs from npx, nothing to install, and validates cheaper or newer models on your own traces.

View on GitHub