Source: https://www.rightmodeler.com/integrations/openrouter

# rightmodeler + OpenRouter

Every candidate call in a rightmodeler audit (shortlist, replay, judge, price) can run through OpenRouter on your own key. It is the engine of the audit, not a source of traces.

rightmodeler talks to any OpenAI-compatible endpoint you name with --base-url, reading the key from the environment variable you name with --api-key-env. Against OpenRouter that means GET /models builds the shortlist from live pricing, context lengths, and supported_parameters, POST /chat/completions runs the replays and the judge, and usage.cost prices each call at what was actually charged rather than a token-count guess.

## How it works with OpenRouter

### 01 · Key: One key, named not passed

Create a key at openrouter.ai and export it yourself. You pass rightmodeler the variable's name, never its value: --api-key-env OPENROUTER_API_KEY. If the variable is unset the run stops and names it rather than falling back to something else, and the key is stripped from any provider error the report or logs ever show.

### 02 · Shortlist: Candidates from the live catalog

GET /models returns every model with pricing, context length, and supported parameters. A candidate qualifies only if it is strictly cheaper than the incumbent on a blended per-token price, supports what the step needs (tool calling, structured output), and has a context window at least as large as the step's observed tokens. The cheapest qualifiers are tried; zero-priced models are excluded unless you pass --include-free.

### 03 · Replay + judge: Real replays, a judge from a third family

Each candidate replays the step's recorded inputs through POST /chat/completions. The built-in judge then makes two temperature-zero calls with the reference and candidate positions swapped, and is chosen from a family that is neither the candidate's nor the incumbent's, so no family grades its own output. Rate limits are retried with backoff and the concurrency cap adapts down on 429s rather than hammering the account.

### 04 · Price: Real prices from usage.cost

Every response carries usage.cost, the amount OpenRouter actually charged, and the report uses that figure rather than token arithmetic across incompatible tokenizers. When a response reports no cost, or returns content with no usage at all, rightmodeler falls back to catalog prices and marks the amount as an estimate instead of presenting it as billed.

## Setup

The audit needs one credential: an OpenRouter key, exported by you and named on the command line. Run estimate first to project the spend before any paid call, and cap the run with --max-cost-usd.

```bash
npx rightmodeler init
```

```bash
# # one key powers the shortlist, every replay, and the judge
export OPENROUTER_API_KEY=sk-or-...
```

```bash
# # project the replay spend before anything is billed
npx rightmodeler estimate --traces ./traces/spans.json --base-url https://openrouter.ai/api/v1 --api-key-env OPENROUTER_API_KEY
```

```bash
# # then run the pipeline under a hard spend cap
npx rightmodeler init --traces ./traces/spans.json --base-url https://openrouter.ai/api/v1 --api-key-env OPENROUTER_API_KEY --max-cost-usd 25
```

## Use cases

### Candidates from every family, one account

The shortlist draws from OpenRouter's whole catalog, so a step sitting on one provider's frontier model gets tested against cheaper models from every family, without you creating an account or holding a key per provider.

### A judge that cannot grade its own homework

Cross-family judging needs a catalog that spans providers. rightmodeler picks a judge from a family different from both the candidate and the model you run today, and if no neutral family is available it fails rather than borrowing one of the two under test.

### Savings measured in dollars, not tokens

Because usage.cost reports the actual charge per call, the report compares what a step costs today against what the candidate charged in replay. No per-token arithmetic across incompatible tokenizers, no stale pricing tables.

### Capability-compatible shortlists

The supported_parameters field becomes a capability filter, so a candidate that cannot do tool calling or structured output never reaches the bench when the step needs one, and a verdict is never flattered by a model that quietly skipped the hard part.

## The honest part

- Replays and judge calls are billed to your key. estimate projects the spend from catalog prices and the corpus before anything runs, and --max-cost-usd stops the run at a ceiling you set, but the money is yours.
- One replay endpoint is configured per run. OpenRouter, the Vercel AI Gateway, and a LiteLLM proxy are all OpenAI-compatible routes rightmodeler accepts; it does not mix keys within a run.
- Zero-priced models are excluded by default: free tiers rate-limit and disappear mid-run, which corrupts a measurement more than it saves. Pass --include-free if you want them counted anyway.
- A step whose current model is missing from the catalog abstains rather than being priced against a guess, and so does a model with no pricing published.
- A cheaper model existing in the catalog is not a recommendation. When position-swapped verdicts disagree, when the evidence is thin, or when the winner cannot clear the quality floor again on held-out cases, the audit abstains.
- rightmodeler calls OpenRouter offline, on traces you exported. It is not a gateway: nothing routes your production traffic through OpenRouter, before or after the audit.

## FAQ

### Do I need an OpenRouter account?

Only if you choose it as the replay route. rightmodeler takes whatever OpenAI-compatible base URL you pass, so the Vercel AI Gateway or your own LiteLLM proxy work the same way. What it always needs is a catalog at /models and a chat endpoint at /chat/completions.

### Why route the audit through OpenRouter?

The shortlist needs one live catalog with prices, context lengths, and capability flags across model families, and the judge needs a family outside both the candidate's and the incumbent's. One OpenAI-compatible endpoint and one key cover all of that; a drawer of per-provider keys covers almost none of it.

### How much will the replays cost me?

Run rightmodeler estimate first: it projects the spend from the corpus and current catalog prices, split between the shortlist half and the held-out half, before any paid call. Then cap the real run with --max-cost-usd, which refuses cases the remaining budget cannot cover rather than overspending.

### Where does my API key live?

In your own environment. You pass the variable's name with --api-key-env and rightmodeler reads it at call time; the value is never written to the store, never echoed, and is stripped from provider error excerpts before anything is logged or rendered.

### Does my trace data leave my machine?

Step inputs go to OpenRouter with your key when a replay runs, the same class of call your agent already makes to a model provider. Nothing goes anywhere else: no rightmodeler server, no account with us, no telemetry.

### If I approve a swap, do I have to serve it through OpenRouter?

No. The pull request changes a model identifier, not a route. You serve the approved model through whatever provider or gateway you already use; OpenRouter was the test bench, not a dependency you ship.