Toward production AI that maintains itself

The human in the loop is the bottleneck.

rightmodeler takes the work out of the loop, not the judgment. It validates cheaper or newer models on your production traces against the outputs you accepted, and opens a draft pull request only when a swap clears every gate. Today, the final yes stays yours.

View on GitHub
Illustrative, not measured results
rightmodeler · per-step swap validation5 steps · quality floor 0.90
Clearedpr_summary
gpt-5.6 → gpt-5.4
save 72%quality 0.94reference+judge
Pending, end-to-end confirmationtask_plannerCASCADE · unconfirmed
claude-opus-5 → llama-3.3-70b
save 41%quality 0.91reference+judge
Clearedjson_extraction
gpt-5.4 → qwen3.7-flash
save 68%quality 1.00deterministic
Clearedsql_generation
gpt-5.4 → deepseek-chat
save 55%quality 0.91reference
Abstainedauth_code_editNO EVIDENCE · abstain
gpt-5.6 → ·
save ·quality ·none
72%

cost reduction · PR summary · illustrative example

Reads the traces you already have.

Autodetects 12 trace formats into one per-step schema.

Reads traces from Claude Code, Codex, LangSmith / LangGraph, OpenAI SDK, Langfuse, Braintrust, Phoenix (OpenInference), OpenTelemetry GenAI, Vercel AI SDK, Helicone, W&B Weave, Bifrost.

Why model swaps stall

Model maintenance is a standing job.

Every few weeks a new model restarts the manual check. Teams take four shortcuts. None holds up.

Blind swaps fail quietly.

Nothing throws. The JSON still parses and is wrong on the edge cases.

Routers don’t work in production.

Our view: unvalidated on your workload, an extra hop, harder to reproduce.

General benchmarks aren’t your workload.

A leaderboard ranks models on someone else’s tasks, not your accepted outputs.

Frontier everywhere costs too much.

It feels safe until the bill compounds on steps nobody has measured.

What holds up: one pinned model per step, validated on your own traces.

A swap is safe when it clears every release gate on your own traces.

What teams say about rightmodeler.

Chris Myers portrait

“rightmodeler took AI Assist from brute force to precision routing. Costs fell 70.8%, responses got twice as fast, and quality held at 100%, measured, not assumed.”

Chris Myers, CEO, B:Side Capital and Fund

Quality measured on a 20-query benchmark against the outputs B:Side had accepted; cost and speed are projected from the same benchmark.

Start with the traces you already have.