Integration · LiteLLM
rightmodeler + LiteLLM
If a LiteLLM proxy already fronts your model traffic, it can be the route the audit replays through. rightmodeler needs an OpenAI-compatible base URL and the name of the variable holding its key, and your proxy is exactly that.
TL;DR
rightmodeler talks to whatever OpenAI-compatible endpoint you name with --base-url, reading the key from the variable you name with --api-key-env, so a LiteLLM proxy is a first-class replay route: GET /models builds the shortlist and POST /chat/completions runs the replays and the judge. It works when your proxy's model list publishes pricing, because the shortlist is built from price, capability, and context window.
How it works with LiteLLM
Name the proxy and the key variable
Start or reuse your gateway, then pass rightmodeler its base URL and the name of the environment variable holding a virtual key: --base-url http://localhost:4000/v1 --api-key-env LITELLM_API_KEY. The value is read at call time, never written to the store, and stripped from provider error excerpts before anything is logged.
The shortlist comes from your model list
GET /models is read as the catalog. A candidate qualifies only when it is strictly cheaper than the incumbent on a blended per-token price, supports what the step needs, and has a context window at least as large as the step's observed tokens. That means the proxy has to publish pricing, context length, and supported parameters for its models; entries without a price cannot be compared and are left out.
Your route, the same evidence contract
Candidates replay the step's recorded inputs through POST /chat/completions on your proxy, and the built-in judge makes two temperature-zero calls with the reference and candidate positions swapped, chosen from a family that is neither the candidate's nor the incumbent's. Rate limits are retried with backoff and the concurrency cap adapts down rather than hammering your gateway.
Coupled steps run in a container
A step whose output can feed later model-authored steps is confirmed by running your pipeline inside a container image you name in a --modeb-config file, with your app mounted read-only. Model calls leave through rightmodeler's own metered proxy, which swaps the model per step, meters every attempt, and refuses a request carrying no correlation header. Your LiteLLM gateway is the replay route, not the swap mechanism.
Setup
There is nothing to install into your app. If you already run a LiteLLM proxy, the audit can use it as the replay route; if you do not, any other OpenAI-compatible endpoint works the same way.
# start (or reuse) the gateway you already operate
# project the replay spend through your proxy first
# then run the pipeline under a hard spend cap
# or let your coding agent drive it
Use cases
Reuse the gateway you already run
If LiteLLM already fronts your model traffic, the audit does not need a second account or a second key. The same virtual key, budgets, and provider routing you already configured apply to the replay and judge calls too.
Keep replay traffic inside your own perimeter
Replays are ordinary chat completions against the base URL you name, so a self-hosted proxy keeps recorded inputs on infrastructure you control rather than sending them to a route you do not operate.
Bring your own provider mix
Because the shortlist is built from whatever your proxy's model list publishes, the candidates are the models you have actually enabled, at the prices you have configured, instead of a public catalog you do not buy from.
The honest part
- rightmodeler has no LiteLLM log adapter. Proxy logs are not a supported trace format; if you want traces from that traffic, export OpenTelemetry GenAI spans and use the OTel adapter, or log OpenAI SDK calls in the JSONL shape that adapter reads.
- The shortlist needs prices. A model list that omits pricing, context length, or supported parameters yields no qualifying candidates, because the audit will not compare models on a price it had to invent.
- Cost comes from what the response body reports. A proxy that returns no cost field leaves rightmodeler computing the amount from catalog prices and token counts, and the ledger labels that figure an estimate.
- One replay endpoint is configured per run. A LiteLLM proxy, OpenRouter, and the Vercel AI Gateway are all OpenAI-compatible routes rightmodeler accepts; it does not mix keys within a run.
- Replays are not bit-exact, even at a fixed temperature. Repeated cases are read as a distribution and the release decision binds on a worst-case lower bound rather than a single sample.
- The proxy is a route, not a hop in production. rightmodeler runs offline on traces you exported and never sits in your request path.
Frequently asked questions
Do I need to run LiteLLM to use rightmodeler?
No. rightmodeler needs an OpenAI-compatible base URL and a key variable name; a LiteLLM proxy is one option, OpenRouter and the Vercel AI Gateway are others. Choose the route you already fund and operate.
Can rightmodeler ingest my LiteLLM proxy logs?
Not directly. There is no LiteLLM trace adapter. The supported paths for that traffic are OpenTelemetry GenAI spans, read by the OTel adapter, or a per-call JSONL log in the OpenAI SDK shape, read by that adapter.
Why does my run find no candidates through my proxy?
Almost always because the model list carries no pricing. The shortlist requires a candidate that is strictly cheaper than the incumbent on a blended per-token price, so a catalog without prices produces nothing to compare. Publish pricing on the proxy's model list, or point the run at a route that already does.
Does rightmodeler replace LiteLLM?
No, they do different jobs. LiteLLM routes live traffic; rightmodeler is an offline audit that borrows a route during replay and never sits in your request path. If a LiteLLM gateway already fronts your traffic, keep it.
How does the end-to-end confirmation swap the model?
Not through your proxy config. Confirmation runs your pipeline in a container and routes its model calls through rightmodeler's own metered proxy, which applies the swap per step, charges every attempt, and fails closed on a request that arrives without correlation metadata rather than guessing which step it belonged to.
Will the end-to-end pass always confirm a swap?
No, and that is why it runs. Confirmation exists to kill shortlisted swaps that cascade: a candidate that drifts mid-pipeline fails here even when its isolated outputs looked fine, and an inconclusive result is a legitimate terminal outcome rather than a pass.
Run the audit on your own traces
The CLI runs from npx, nothing to install; your LiteLLM traces do the rest.