Comparison · Vercel AI Gateway

rightmodeler vs Vercel AI Gateway

The Gateway sits in your request path and optimizes everything after the model name in the request. rightmodeler runs before any change: it replays the steps you already shipped to ask whether a cheaper or newer model could take that name's place.

Complement · same credits, two jobsVisit Vercel AI Gateway  (opens in a new tab)

TL;DR

You hire the Vercel AI Gateway to serve model traffic: one key to hundreds of models, automatic retries when a provider fails, and a live meter on every request's cost, latency, and tokens. You hire rightmodeler for the question that meter never asks: could a cheaper or newer model do the job of the one named in each request? rightmodeler replays steps you already shipped through cheaper or newer candidates, judges them against the output you accepted, reports the evidence, and opens a draft pull request when a swap clears every gate. The two compose: the Gateway optimizes everything after the model name; rightmodeler questions the name itself.

The Gateway alone: a perfect meter

  • Spend, requests, and latency, attributed per project and per API key, with a per-request log showing the exact fallback path.
  • Routing picks providers by recent uptime and latency, and you can sort by cost, time to first token, or throughput.
  • The model name in the request is yours. Nothing in the Gateway scores output quality or asks whether a cheaper model would have sufficed.

With rightmodeler on top

  • Your shipped steps replay through cheaper or newer candidates, offline, so production never sees an experiment.
  • A judge from a neutral third model family scores each replay against the output you accepted: reference evidence (agreement with what you shipped, not ground-truth correctness).
  • A quality floor (a configurable minimum agreement score) guards every recommendation, and thin evidence ends in an abstention (no recommendation at all) instead of a guess.

The meter and the measurement

What each one watches, when it acts, and what it can never tell you.

vercel-ai-gateway vs rightmodeler
watches
Vercel AI Gateway · live traffic: requests, tokens, spend, latency
rightmodeler · recorded steps: the traces you already shipped
acts
Vercel AI Gateway · during the request: routing, retries, fallbacks
rightmodeler · before the change: replays, judging, a report and a draft pull request you review
optimizes
Vercel AI Gateway · the route for the model your code names
rightmodeler · the model name itself, one step at a time
can say
Vercel AI Gateway · who spent what, and which provider served it
rightmodeler · which steps could move to a cheaper or newer model, with evidence, sample size, and abstentions
can never say
Vercel AI Gateway · whether a cheaper model would have sufficed
rightmodeler · anything about a request in flight; it is offline by design

Which one gets the call?

Three situations, each with the honest winner.

A provider has an outage during your launch demo and requests start failing.

the right hire: Vercel AI Gateway

This is a live request that needs saving right now. The Gateway retries across providers automatically, and can apply per-provider timeouts and fall back to backup models you declare. rightmodeler is offline by design and cannot help a request in flight.

Finance asks which project and which API key spent, say, 400 dollars on inference last week.

the right hire: Vercel AI Gateway

That is a metering question, and the meter is theirs: the dashboard attributes spend, requests, and latency per project and per API key, and every request lands in a searchable log. rightmodeler has no runtime metering at all.

You already serve production through the Gateway and want the bill lower without breaking quality.

the right hire: both, together

rightmodeler spends the Gateway credits you already fund, replays your real steps through cheaper or newer candidates from the Gateway's live catalog, and a swap that clears every gate becomes a draft pull request that changes only model identifiers, still served through the Gateway once you merge it. Nothing about your runtime changes except the name.

Same credits, two jobs

A shipped integration makes the Gateway rightmodeler's replay endpoint.

Set your key and rightmodeler runs on the Gateway you already fund: the candidate shortlist comes from the live model catalog, so it never goes stale; your key pays for both the replays and the judging; and each call is priced from what the Gateway reports. A swap at a step whose output feeds later model steps counts only once it is confirmed end to end in a sandbox, because cascade risk (one step's cheaper output quietly degrading the steps after it) hides from a single-shot test. The sandbox runs your app and keeps your model key out, but it has ordinary network access, so point your app's tools at test fixtures, not production services.

When a swap clears every gate, the draft pull request changes only the model name, and once you merge it the Gateway keeps serving it. For a validation run the Gateway is a bench you borrow, not a dependency you ship.

# rightmodeler replays through the Gateway you already fund

# then run rightmodeler on your exported traces

Setup guide: rightmodeler + Vercel AI Gateway

The honest part

  • The Gateway serves live traffic and keeps it up. Automatic retries across providers, per-provider timeouts, and model fallbacks handle an outage in real time. rightmodeler never serves a single production request.
  • The Gateway meters and controls spend at runtime: a live per-request log, per-project and per-key attribution, and budgets that stop a request before it runs once a cap is hit. rightmodeler only prices the replays in a validation run.
  • The Gateway is the model access itself: one key to hundreds of models, zero markup on tokens, and you can bring your own provider keys. rightmodeler brings no models; it spends your Gateway credits to test them.
  • And rightmodeler's verdicts are hedged on purpose: they establish that a candidate agreed with the output you shipped, and nothing stronger. The production result is the reference, not ground truth.

Terms this page leans on

reference evidence
How a replay is scored: agreement with the output you actually accepted for that exact step. The production result is the reference, not ground truth.
quality floor
The configurable minimum agreement score a candidate must clear before rightmodeler recommends it for a step.
abstain
What rightmodeler does when a step's evidence is too thin to support a verdict: it says so and recommends nothing.
cascade risk
The danger that a cheaper model's slightly different output at one step degrades the steps downstream of it. End-to-end confirmation in a sandbox exists to surface it.

Frequently asked questions

Does rightmodeler replace Vercel AI Gateway?

No. The Gateway serves and meters your live traffic; rightmodeler validates cheaper or newer models on traces of traffic already served. Not observability. Not a runtime gateway. It never sits in your request path. A validation run actually gives the Gateway more work: replays run through it, and a merged swap keeps serving through it.

The Gateway already shows cost per model. Why replay anything?

The dashboard tells you what each model spent, not whether a cheaper model would have produced the output you accepted. rightmodeler reruns those same steps on cheaper or newer candidates and reports, per step, how closely each one matched what you shipped, with sample size and abstentions attached. The meter prices your model decision; it was never built to make it.

Can't I just sort providers by cost in the Gateway?

Provider sorting changes who serves the model you named; it never changes the model. Budgets and routing act on the route and the cap, not on model choice or quality. Nothing in the Gateway scores output quality; asking whether a cheaper or newer model would have sufficed is rightmodeler's whole job.

Do I need a Gateway account to run rightmodeler?

No. The Gateway is one of the supported replay endpoints, not a requirement; rightmodeler can replay through another supported provider instead. But if you already fund Gateway credits, the same balance powers the replays, and the report stays a recommendation of a model, not a route.

Will rightmodeler always find a cheaper or newer model?

No. Plenty of steps are already on the cheapest model that can do the work, and a candidate that survives the capability checks can still fall short of the quality floor, which ends with no recommendation, or leave the judge with too little evidence, which ends in an abstention. And a swap that clears every gate is only a draft pull request until a person merges it.

Start with the traces you already have.

The CLI runs from npx, nothing to install, and validates cheaper or newer models on your own traces.

View on GitHub