How it works
How rightmodeler works
Detect, measure, review. The whole loop, run on the traces you already have.
TL;DR
rightmodeler replays your real agent traces through cheaper models, judges each output against what you already shipped, then reports the evidence, sample size, and abstentions for your review.
in ▸ raw traces
- 01
Detect
Point it at the traces you already emit. rightmodeler autodetects the format across 9 sources and folds every run into one per-step schema, with no new SDK and no re-instrumentation.
reads Claude Code · Codex · LangSmith / LangGraph · OpenAI SDK · Langfuse · Braintrust · Phoenix (OpenInference) · OpenTelemetry GenAI · LiteLLM StandardLoggingPayload → 1 per-step schema
- 02
Measure
It replays each step through cheaper candidates on your real inputs and measures every output against what you accepted. Each candidate gets a cost delta, reference-agreement score, evidence count, and risk flag, and it abstains when the evidence is weak.
scores cost · agreement · evidence count · risk flag → recommendation + confidence · abstain on high-risk
- 03
Review
You review each recommendation and rightmodeler applies only the edits you approve via the Skill. A report and an edit, never a live intercept. You decide what to change, and when.
applies approved model edit in your repo, via the Skill · nothing changes without your approval
out ▸ signed report
How to read confidence.
Hard checks run before a model judge. When judgment is needed, a cross-family judge scores both output orders. Evidence counts show what earned the confidence band, and the evidence type limits how high that band can go. Confidence applies only to the prompt, inputs, and runs evaluated. It measures agreement with what you shipped, not proof of correctness.
Not observability. Not a runtime gateway.
Observability only shows you problems; a gateway hijacks live traffic. rightmodeler measures candidates on runs you already shipped, then applies only the edits you approve.
FAQ
- Which traces are supported?
- 9 formats, autodetected: Claude Code, Codex, LangSmith / LangGraph, OpenAI SDK, Langfuse, Braintrust, Phoenix (OpenInference), OpenTelemetry GenAI, and LiteLLM StandardLoggingPayload. rightmodeler folds them all into one per-step schema, so you point it at the traces you already emit, with no new instrumentation.
- Does it touch production?
- No. rightmodeler replays your past traces offline and produces a report plus a repo edit. It never sits in your request path, routes live traffic, or adds latency. It is not a runtime gateway.
- Do you store my data?
- It runs locally on your own traces and your own replay provider key. Replays call your selected provider, OpenRouter, the Vercel AI Gateway, or a LiteLLM proxy, using your key; there is no rightmodeler server holding your traces.