Integration · Vercel AI Gateway
rightmodeler + Vercel AI Gateway
The Vercel AI Gateway is one of the OpenAI-compatible routes rightmodeler replays through. Candidate calls and the judge run on your key, against models discovered from the Gateway's live catalog.
TL;DR
rightmodeler takes any OpenAI-compatible endpoint with --base-url and reads its key from the variable you name with --api-key-env. Against the Gateway that means GET /models supplies per-token pricing, context windows, and capability flags for the shortlist, POST /chat/completions runs the replays and the judge, and the cost reported on each response is recorded as billed, with catalog pricing used as a labelled estimate when none comes back.
How it works with Vercel AI Gateway
Name the variable, not the key
Export your Gateway key yourself and pass rightmodeler its variable name: --api-key-env AI_GATEWAY_API_KEY. The value is read at call time, never written to the store, and stripped from provider error excerpts before anything is logged. An unset variable stops the run and names the variable rather than silently trying something else.
Discover candidates from the live catalog
GET /models supplies per-token input and output pricing, context windows, and capability tags such as tool use and structured output. rightmodeler normalizes those fields, including the Gateway's context_window naming, and builds the shortlist at run time from models strictly cheaper than the incumbent that still cover what the step needs, so there is no fixed model list to go stale.
Replay the recorded step, judge across families
Each candidate replays the step's recorded inputs through POST /chat/completions, with the output limit floored at the smallest value the Gateway accepts so a short recorded answer is not rejected outright. The built-in judge then makes two temperature-zero calls with the reference and candidate positions swapped, chosen from a family that is neither the candidate's nor the incumbent's; if no neutral family can be resolved the run fails rather than borrowing one.
Use the Gateway's charge for each call
rightmodeler records the cost the response reports, preferring the billed amount and falling back to the market or upstream figure the Gateway includes. A billed zero is not read as free. When no cost comes back, or content arrives with no usage at all, the amount is computed from catalog pricing and token counts and marked as an estimate in the ledger.
Setup
Use a Vercel AI Gateway key with credits, exported by you and named on the command line. Project the spend with estimate before the first paid call, then cap the real run.
# replay and judge calls bill to your Gateway credits
# project the replay spend before anything is billed
# then run the pipeline under a hard spend cap
# or let your coding agent drive it
Use cases
Audit on the route you already fund
If your team already has Gateway credits, the same account can fund candidate replays and judge calls, and estimate tells you roughly what the run will draw down before it starts.
Test against a catalog that stays current
The shortlist comes from the Gateway's live /models response, including current pricing, context windows, and capability tags. New and retired models change the next run without a rightmodeler release.
Keep capability checks in the shortlist
Tool-use and structured-output tags become capability filters, and the context window is checked against the step's observed tokens before a candidate reaches replay. A lower price alone is not enough to make the bench.
Survive a rate limit without corrupting the run
A 429 halves the concurrency cap and the request is retried with backoff, honoring the Gateway's retry-after when it sends one. A rate limit is a transport event, so it is never recorded as an execution outcome the statistics would count.
The honest part
- Replay and judge calls spend your Gateway credits. estimate projects the spend from catalog prices and the corpus first, and --max-cost-usd caps the run, but the evidence is produced with real billable calls.
- Cost is only as good as what the response reports. When the Gateway returns no cost for a call, rightmodeler falls back to token usage and catalog pricing and labels that amount an estimate rather than presenting it as billed.
- A model the catalog publishes with no pricing cannot be shortlisted, and a step whose current model is missing from the catalog abstains rather than being compared against a guess.
- A cheaper candidate is not automatically a recommendation. Capabilities are filtered first, thin or inconsistent evidence ends in abstention, and the winner has to clear the quality floor again on held-out cases.
- One replay endpoint is configured per run. The Gateway, OpenRouter, and a LiteLLM proxy are all OpenAI-compatible routes rightmodeler accepts; it does not mix keys within a run.
- rightmodeler is an offline report on traces you provide, never a runtime gateway or a hop in your production request path. Replay inputs still leave your machine for the Gateway when the audit calls a model.
Frequently asked questions
Do I need a Vercel AI Gateway account?
Only if you choose it as the replay route. rightmodeler takes whatever OpenAI-compatible base URL you pass, so OpenRouter or your own LiteLLM proxy work the same way. What it always needs is a catalog at /models and a chat endpoint at /chat/completions.
Why route the audit through the Gateway?
It gives rightmodeler one OpenAI-compatible route plus a live model catalog with pricing, context windows, and capability tags. That is enough to shortlist cheaper capable models, replay them, and choose a judge from a neutral family without a fixed model list.
How are replay costs measured?
From the cost the response reports, preferring the billed amount and falling back to the market or upstream figure when the billed one is absent or zero. If none is present, the amount is computed from token usage and current catalog prices and marked as an estimate.
How do I know what a run will cost before it runs?
Run rightmodeler estimate. It projects the spend from the corpus and current catalog prices, split between the shortlist half and the held-out half, before any paid call is made. Then cap the real run with --max-cost-usd, which refuses cases the remaining budget cannot cover.
Does my trace data leave my machine?
The audit and report run locally, but the recorded step inputs used for replays and judging are sent to the Gateway with your key. There is no rightmodeler server, and the audit adds nothing to your production request path.
If I approve a swap, must I keep using the Gateway?
No. The pull request changes a model identifier, not a route. The Gateway is the test bench for this audit; you can serve an approved model through whichever provider or gateway fits your application.
Run the audit on your own traces
The CLI runs from npx, nothing to install; your Vercel AI Gateway traces do the rest.