Comparison · LiteLLM
rightmodeler vs LiteLLM
LiteLLM decides in the milliseconds before each call; rightmodeler decides after the run, from replays of outputs you already accepted. Keep the proxy: the audit can replay through it, on your own keys and budgets.
TL;DR
Keep your proxy. LiteLLM is hired to run the request path: one OpenAI-compatible API in front of the model catalog, budgets that hard-stop overruns, retries and fallbacks when a provider degrades. rightmodeler is hired for a different job on a different clock: after the fact, it replays the agent traces (the recorded runs) you already shipped through cheaper models and reports, step by step, which swaps hold up. Nothing about it touches your request path: the audit is a document you review, not a hop your traffic takes. And the proxy you already run can be the audit's replay route: candidate and judge calls go through it, on your own keys and budgets.
Your proxy as the audit's replay route
rightmodeler replays and judges through any OpenAI-compatible base URL, and a LiteLLM proxy is one.
Replay route. Point --base-url at the proxy's /v1 root and --api-key-env at the variable holding a virtual key, and the audit's shortlist, replay, and judge calls run through the gateway you already operate, on your own keys, budgets, and routes. There is no rightmodeler server, no account with us, no telemetry.
Catalog. LiteLLM's /v1/models lists model ids only, so pass --catalog-reference with the upstream's public model list and name each model group by its upstream id, such as openai/gpt-4o-mini: the reference fills in each listed model's price, context window, and capabilities. When no model is priced after that join, rightmodeler reads per-token costs from the proxy's GET /model/info. Give each replay model a group of its own, with no fallbacks, aliases, or caching, so every answer comes from the model it names.
Traces. rightmodeler has no LiteLLM log adapter. For traffic through the proxy, export OpenTelemetry GenAI spans, or log OpenAI SDK calls as JSONL, and the matching adapter reads them. The LiteLLM integration page has the full setup.
# replay and judge through your proxy, with the upstream's public model list for price and context
LiteLLM alone: a governed request path
- The router decides in the milliseconds before each call, from operational signals: rate-limit headroom, latency, cost, with retries, cooldowns, and fallbacks across providers.
- Budgets and rate limits hard-stop a key, team, or user once it crosses its cap. The bill gets a ceiling.
- Every call lands in one standardized log. You can see exactly what you spent; the log cannot say whether a cheaper model would have done the job.
With rightmodeler on top
- Traces of the same calls become audit input: the steps you already paid for, rerun on cheaper candidates through your proxy and scored against the answers you accepted.
- Every recommendation carries reference evidence (agreement with the output you shipped, not proof of correctness), a sample size, and abstentions (steps where the audit declines to rule because the evidence is thin).
- You approve every swap, and the report recommends a model, not a route: serve the approved model through the proxy you already run.
Three situations, called honestly
Who wins depends on the job. Here is the straight answer for each.
A platform team must give dozens of internal teams LLM access, each with a monthly budget that hard-stops any team that overruns.
the right hire: LiteLLM
This is governance inside the request path, and it is LiteLLM's home turf: budgets set at proxy, team, key, and user level fail requests once a cap is crossed. rightmodeler has no keys, no budgets, and no enforcement of any kind.
Your LiteLLM spend logs show the agent's bill doubled, and you want to know which of its steps could run on a cheaper model without losing quality.
the right hire: rightmodeler
Budgets cap the bill; they do not right-size it. rightmodeler replays your real traces through cheaper candidates and judges each step against the output you accepted, reporting agreement, sample size, and abstentions for every step. No gateway setting answers this question.
You already run a LiteLLM proxy, and your application's model calls are already traced with OpenTelemetry.
the right hire: both, together
The traces are the evidence and the proxy is the route. rightmodeler reads GenAI spans that carry message content, replays recorded steps through cheaper candidates on the same proxy, and judges them against the outputs you shipped. The gateway that runs your traffic carries the audit that right-sizes it.
Different clocks, side by side
The disagreement is not engineering quality. It is when each tool decides and what it can know at that moment.
The honest part
- rightmodeler never sits in your request path, so it cannot retry, fail over, load-balance, cache, or rate-limit anything at runtime. That is LiteLLM's core job, and an audit does not replace a single piece of it.
- rightmodeler also does not enforce spend or front providers: no virtual keys, no budgets, no hard stops, no OpenAI-compatible endpoint in front of LiteLLM's provider catalog, no guardrails, no per-team access control. If that layer is what you are shopping for, LiteLLM is the right choice, plainly.
- The one real overlap is model selection. LiteLLM's auto-router classifies each incoming prompt into a model tier at request time; it ships today and takes minimal setup: a small config block in the proxy you already run. The honest caveat is the evidence: it reads the question, never the outcome, so it cannot tell you whether the cheap tier's answers hold up. rightmodeler measures replays against outputs you accepted, and an audit's findings can inform the very tier map you hand that auto-router.
- The measurement cuts both ways. A candidate that matches what you shipped has matched your production behavior and nothing more: the production result is the reference, not ground truth. When the sample is too small to support a verdict, the audit abstains instead of guessing.
Frequently asked questions
Do I drop LiteLLM once rightmodeler is running?
No. LiteLLM runs your request path; rightmodeler decides, with per-step evidence, what belongs in it. An audit recommends models, never routes: you approve a swap and keep serving it through the proxy you already run. Crucible, the optimization suite on the roadmap, is designed to hand routing back to a gateway you bring, LiteLLM included.
LiteLLM's auto-router already picks models per request. Why measure offline?
Because of what a single request can know. When Bai et al. reran eight frontier models over 500 real GitHub issues, four repeats apiece with the task held identical, token usage still swung up to thirtyfold between repeats (How Do AI Agents Spend Your Money?, arXiv:2604.22750). A classifier reading one prompt before the answer exists is guessing inside that variance. rightmodeler measures across replays of outputs you accepted, recommends only candidates that clear your quality floor (an agreement bar below which no swap is offered), and abstains when the sample is thin.
Can an audit start from the logs my proxy already writes?
Not from the proxy's own logs: rightmodeler has no LiteLLM log adapter. It reads OpenTelemetry GenAI spans or OpenAI SDK JSONL of the same calls, and it can replay through the same proxy. The LiteLLM integration page walks through the full setup.
Does an audit ever touch my production traffic?
No. rightmodeler is a report you run on traces you already have, offline. Replay calls go to the provider you choose, your own proxy included, and nothing destructive runs against your real repo: code execution happens in a disposable copy with side-effecting tools mocked from the recorded trace unless you opt into live execution. You approve every swap before anything changes.
Run the audit on your own traces
The CLI runs from npx, nothing to install, and your own traces settle the question.