# rightmodeler

> rightmodeler keeps AI agents on the right model at every step. The rightmodeler CLI, published on npm as rightmodeler, replays your real agent traces through cheaper models, measures each candidate against the output you accepted, reports the evidence, sample size, and abstentions, and ships approved swaps as a pull request you can roll back byte-exact. npx rightmodeler init finds the traces Claude Code and Codex already left on disk and runs free through shortlist; a coding-agent skill lets Claude Code and Codex-class agents drive the same CLI end to end. rightmodeler agent, open source in the same repo with a hosted version coming, watches prices, drift, and new model releases and opens evidence-backed model-change pull requests in your repo. Crucible, in early access, is the analytics and optimization suite that shows what every layer of your agent system costs, how fast it runs, and where it fails, and keeps your stack right-sized continuously. The CLI produces a report you run on your own traces, not a runtime gateway.

rightmodeler is a developer tool from ELM-OS for teams running multi-agent LLM systems. This file indexes the site for language models; for a curated product overview, annotated page index, and complete blog archive in one file, see the Optional section.

## Product

- [rightmodeler](https://www.rightmodeler.com): Home. What it does, how the evidence-backed downgrade report works, and the first command to run.
- [GitHub repository](https://github.com/elm-os/rightmodeler): Source code and the pipeline you run on your own traces. Open source under the MIT license.

## Developer resources

- [CLI documentation](https://github.com/elm-os/rightmodeler/tree/main/harness/packages/rightmodeler/docs): Command reference, getting started, exit codes, evaluators, and Mode B. Ships with the npm package.
- [Getting started](https://github.com/elm-os/rightmodeler/blob/main/harness/packages/rightmodeler/docs/getting-started.md): Install, point it at traces, and read the first report.
- [Command reference](https://github.com/elm-os/rightmodeler/blob/main/harness/packages/rightmodeler/docs/commands.md): Every command and flag, generated from the CLI itself.
- [Exit codes](https://github.com/elm-os/rightmodeler/blob/main/harness/packages/rightmodeler/docs/exit-codes.md): What each exit status means, for scripting and CI.
- [Evaluators](https://github.com/elm-os/rightmodeler/blob/main/harness/packages/rightmodeler/docs/evaluators.md): Scoring replays with Braintrust, Langfuse, LangSmith, or promptfoo instead of the built-in judge.
- [Coding-agent skill](https://github.com/elm-os/rightmodeler/blob/main/skills/rightmodeler/SKILL.md): The runbook that lets Claude Code and Codex-class agents drive the CLI end to end. Install with npx skills add elm-os/rightmodeler --skill rightmodeler
- [npm package](https://www.npmjs.com/package/rightmodeler): The published CLI. Run it with npx rightmodeler init
- [Source repository](https://github.com/elm-os/rightmodeler): The pipeline, the agent, and the schemas. MIT licensed.

## Pages

- [How it works](https://www.rightmodeler.com/how-it-works): Detect, measure, review: how rightmodeler replays your traces through cheaper models, measures each output against what you accepted, and reports evidence, sample size, and abstentions.
- [Reduce LLM costs](https://www.rightmodeler.com/use-cases/reduce-llm-costs): Cut your agent's model bill without guessing: measure cheaper candidates against accepted outputs and review the evidence before changing a model.
- [Case studies](https://www.rightmodeler.com/case-study): How real teams modeled lower inference cost with per-workload routing policies; B:Side also reports a measured 100% pass rate on a 20-query benchmark.
- [Case study: B:Side Assist](https://www.rightmodeler.com/case-study/bside): How rightmodeler right-sized AI Assist's 11 AI layers: 70.8% lower projected inference cost and a measured 100% pass rate on a 20-query benchmark.
- [Case study: iAM360](https://www.rightmodeler.com/case-study/iam360): How iAM360 used rightmodeler to cut modeled AI cost per request by 56-57% while upgrading its hardest coaching paths from Terra to Sol.
- [Integrations](https://www.rightmodeler.com/integrations): Every tool rightmodeler works with: the trace formats it reads and the infrastructure it replays through.
- [rightmodeler vs alternatives](https://www.rightmodeler.com/vs): Honest comparisons with routers, gateways, and eval platforms: what each tool decides, what it measures, and when to use which.
- [Manifesto](https://www.rightmodeler.com/manifesto): Measure it, don't guess: the case for evidence-backed model downgrading.
- [Glossary](https://www.rightmodeler.com/glossary): Plain definitions for the model-downgrade vocabulary: quality floor, cascade risk, abstain, LLM-as-judge, and more.
- [rightmodeler agent (coming soon)](https://www.rightmodeler.com/agent): The autonomous agent that watches new model releases, replays them against your real traces, and opens evidence-backed model-swap pull requests in your repo.
- [Crucible (coming soon)](https://www.rightmodeler.com/crucible): The analytics and optimization suite for AI agents: cost per layer, speed per step, failures as they happen, and continuous right-sizing, connected over MCP.
- [About](https://www.rightmodeler.com/about): What rightmodeler is and the mission behind it.
- [Contact](https://www.rightmodeler.com/contact): Every way to reach the rightmodeler team: email for questions and licensing, GitHub issues for bugs, the feedback form for product input, and the security policy for vulnerability reports.
- [Feedback](https://www.rightmodeler.com/feedback): Send the team feedback: what to build, what broke, and what the agent should handle next.
- [Privacy policy](https://www.rightmodeler.com/privacy): What the site collects (very little), how it is used, and how to reach us.
- [Terms of service](https://www.rightmodeler.com/terms): The terms that govern use of the rightmodeler website, in plain English.

## Integrations

- [rightmodeler + Claude Code](https://www.rightmodeler.com/integrations/claude-code): Point rightmodeler at ~/.claude/projects: it replays every session turn through cheaper models and measures each against the output you accepted.
- [rightmodeler + Codex CLI](https://www.rightmodeler.com/integrations/codex): Run rightmodeler on Codex sessions to replay each turn through cheaper models and judge every output against the result you accepted.
- [rightmodeler + LangSmith](https://www.rightmodeler.com/integrations/langsmith): Export a LangSmith run tree: rightmodeler replays each step through cheaper models and measures every output against what you accepted.
- [rightmodeler + the OpenAI SDK](https://www.rightmodeler.com/integrations/openai-sdk): Point rightmodeler at your OpenAI SDK request log: it replays every call through cheaper models and measures each output against what you accepted.
- [rightmodeler + Langfuse](https://www.rightmodeler.com/integrations/langfuse): Export Langfuse observations to disk and rightmodeler replays each step through cheaper models, judging every candidate against the output you already shipped.
- [rightmodeler + Braintrust](https://www.rightmodeler.com/integrations/braintrust): Fetch Braintrust span rows: rightmodeler replays each step through cheaper models and measures every candidate against the output you accepted.
- [rightmodeler + Phoenix](https://www.rightmodeler.com/integrations/phoenix): Export OpenInference spans from Phoenix: rightmodeler replays each step on cheaper models and measures every output against what you accepted.
- [rightmodeler + OpenTelemetry](https://www.rightmodeler.com/integrations/otel): Point rightmodeler at OTLP span exports with gen_ai.* attributes. It replays each step on cheaper models and measures candidate agreement with accepted output.
- [rightmodeler + Helicone](https://www.rightmodeler.com/integrations/helicone): Export Helicone request rows: rightmodeler replays each recorded call against cheaper candidates and scores every one against the response you already shipped.
- [rightmodeler + W&B Weave](https://www.rightmodeler.com/integrations/weave): Export W&B Weave calls: rightmodeler replays each recorded call against cheaper candidates and scores every one against the output you already shipped.
- [rightmodeler + OpenRouter](https://www.rightmodeler.com/integrations/openrouter): Point rightmodeler at OpenRouter with one base URL and one key: it shortlists cheaper models from the live catalog, replays your steps, and prices every call.
- [rightmodeler + LiteLLM](https://www.rightmodeler.com/integrations/litellm): Point rightmodeler at a LiteLLM proxy as its replay route: candidate calls and the judge run through the gateway you already operate, on your own keys.
- [rightmodeler + Vercel AI Gateway](https://www.rightmodeler.com/integrations/vercel-ai-gateway): Point rightmodeler at the Vercel AI Gateway: it finds cheaper candidates in the live catalog, replays real steps, judges quality, and records reported costs.

## Comparisons

- [rightmodeler vs OpenRouter](https://www.rightmodeler.com/vs/openrouter): OpenRouter routes live traffic and, on Auto, picks models from community spend. rightmodeler replays your traces against outputs you accepted. They compose.
- [rightmodeler vs LiteLLM](https://www.rightmodeler.com/vs/litellm): LiteLLM runs your request path: one API, budgets, fallbacks. rightmodeler replays your traces offline to measure which model belongs at each step. Keep both.
- [rightmodeler vs Vercel AI Gateway](https://www.rightmodeler.com/vs/vercel-ai-gateway): Vercel AI Gateway serves and meters your live model calls. rightmodeler replays your traces to test cheaper models first. Complement: same credits, two jobs.
- [rightmodeler vs Braintrust](https://www.rightmodeler.com/vs/braintrust): Braintrust holds your agent to evals you author. rightmodeler checks cheaper models against the outputs you shipped, step by step. They compose.
- [rightmodeler vs LangSmith](https://www.rightmodeler.com/vs/langsmith): LangSmith watches, scores, and runs agents in production. rightmodeler reads its exported run trees and decides which model each step needs. Different job.
- [rightmodeler vs promptfoo](https://www.rightmodeler.com/vs/promptfoo): promptfoo tests what you ship next against cases you author. rightmodeler audits what you shipped against accepted outputs. Different stages of the same loop.
- [rightmodeler vs Not Diamond](https://www.rightmodeler.com/vs/not-diamond): Not Diamond predicts the best model per request, live. rightmodeler measures which model belongs at each step, offline, from your traces. Different clocks.
- [rightmodeler vs Martian](https://www.rightmodeler.com/vs/martian): Martian is a research lab with a name-the-model gateway; its router is legacy. rightmodeler replays traces offline to pick each step's model. Different jobs.

## Blog

- [Blog index](https://www.rightmodeler.com/blog): All posts.
- [Nobody can predict the bill.](https://www.rightmodeler.com/blog/nobody-can-predict-the-bill): Eight frontier models, 500 real GitHub issues, four runs each. A new paper finds nobody can predict what an agent task costs, including the agent.
- [The bill nobody can read.](https://www.rightmodeler.com/blog/the-bill-nobody-can-read): Why your agent bill has no line items, and what to do about it. Part one of the rightmodeler vision: Crucible for AI agent analytics and optimization.
- [The Tuesday problem.](https://www.rightmodeler.com/blog/the-tuesday-problem): Why evaluating every new model becomes a project nobody budgets. Part two of the rightmodeler vision: turning model migrations into pull requests.
- [We were picking models by vibes. Then the bill arrived.](https://www.rightmodeler.com/blog/why-we-built-rightmodeler): We picked models by instinct for years, then defaulted to the frontier model and paid for it. rightmodeler measures candidates against outputs you accepted.

## Optional

- [Curated site context](https://www.rightmodeler.com/llms-context.txt): Product overview, annotated page and integration index, and complete blog posts in one Markdown file.