Comparison · Modaic
rightmodeler vs Modaic
Modaic works on the judge itself: typed decisions, confidence scores, and expert feedback that refines its instructions. rightmodeler puts a judgment to one use, deciding which model each step of your agent needs, measured offline against the outputs you accepted.
TL;DR
Complements, doing different jobs. You hire Modaic to make judgment-heavy decisions consistent: per its site, you define a decision, Modaic surfaces the cases its model is unsure about, experts give feedback, and alignment refines the model, with a confidence score on each decision to decide which ones a person should review. You hire rightmodeler, the free MIT-licensed CLI, for a decision Modaic is not built to make: which model each step of your agent needs. It replays your recorded traces through cheaper candidates offline, grades each one against the output you accepted, and reports reference agreement, sample size, and abstentions; a swap reaches your code only as a draft pull request a human reviews. There is no Modaic integration today. One is planned against Modaic's hosted API, so that a judge you keep in your Modaic account can grade the replays, and it is waiting on Modaic confirming its canonical API; the integration page says exactly what exists and what does not.
Modaic makes the judge trustworthy
- Per its docs, you ask typed questions (true or false, a choice among options, or a score on a scale) and get back typed answers with probabilities, plus a confidence value for choices and scores.
- Confidence decides what reaches a person: the docs suggest choosing a threshold and sending decisions below it to a human reviewer.
- Expert feedback becomes alignment. Annotated examples drive a run that rewrites the model's question instructions, not its weights, and commits a new version only when it beats the starting one.
- Every model lives in a versioned repository, and a commit SHA pins an exact revision for reproducible decisions.
rightmodeler decides which model each step needs
- It reads the traces your agent already recorded, and the output you accepted at each step becomes the reference.
- It shortlists cheaper candidates from your provider's live catalog, replays each step through them, and grades every candidate against that reference.
- Each verdict carries reference agreement, sample size, and abstentions. A winner must clear the quality floor again on held-out cases, and multi-step swaps are confirmed end to end before they are recommended.
- rightmodeler apply opens a draft pull request that changes model identifiers and nothing else. A human reviews it and merges it; the CLI never does.
Same word, different jobs
Both talk about judges, confidence, and human review. What each one points them at is different.
Which one do you hire?
Three situations, with the honest answer in each.
Your trust and safety team reviews thousands of moderation calls a day and wants a model to take the clear ones, sending only the unsure cases to people.
the right hire: Modaic
That is the loop Modaic describes: typed decisions with a confidence value, a threshold that routes the uncertain ones to a reviewer, and expert feedback that refines the model's instructions over time. rightmodeler makes no production decisions and routes nothing to anyone at runtime.
Your agent has run in production for months with one expensive model on every step, and you want to know which steps a cheaper model can hold.
the right hire: rightmodeler
Modaic's workflow starts from a decision you define, not from your agent's recorded steps. rightmodeler starts from the traces: it reruns each step on cheaper candidates, grades them against the answers you shipped, abstains where the evidence is thin, and, when you run apply, opens a draft pull request for your team to review.
Your team already writes its judgment calls as Modaic judges and wants the audit graded by one of them.
the right hire: both, together
Planned, not shipped. The integration is planned against Modaic's hosted API, so a judge you keep in your Modaic account would grade the replays. Until it exists, keep Modaic for the decisions it makes, and grade the audit with the built-in judge or with Braintrust, Langfuse, or LangSmith through --evaluator.
The honest part
- Modaic works on a problem rightmodeler does not touch. Its stated aim is to make AI judgment "auditable, calibrated, and trusted at scale": confidence estimation, review of the uncertain cases, and alignment runs that only commit when they beat the baseline. Per its site it is SOC 2 Type II compliant and offers managed, VPC, or bring-your-own-cloud deployment. rightmodeler has none of that. It is a report you run on your own machine, with no server, no account with us, and no telemetry.
- Where they meet is the judge. rightmodeler already grades replays with a built-in judge from a model family that neither compared model belongs to, or with an external evaluator you choose. A Modaic evaluator is planned against Modaic's hosted API with your own Modaic account, because Modaic's judges, confidence scores, and alignment run only on its service. Each graded case would go to Modaic without naming the candidate model, and a confidence number would never decide whether a candidate passes.
- And the audit's own hedge, whichever judge grades it: a passing score means the candidate agreed with the output you shipped, not that either answer was correct. Where the evidence is thin, the audit abstains instead of guessing.
Terms this page uses
- LLM-as-judge
- A model that grades another model's output. Modaic works on making such a judge consistent and calibrated; rightmodeler uses one, from a family neither compared model belongs to, to grade agreement with the output you accepted.
- confidence score
- Modaic's per-decision estimate of how sure its model is. Its docs call it a ranking and thresholding signal, not a substitute for measuring agreement against labeled examples.
- reference evidence
- Agreement between a candidate's answer and the output you already accepted for the same step. Evidence of agreement with shipped output, not proof of correctness.
- quality floor
- The minimum agreement a candidate must clear before the audit recommends it. The winner has to clear it again on held-out cases.
- abstain
- What the audit does when a step's sample is too small or its evidence too mixed: it recommends nothing rather than guessing.
Frequently asked questions
Does rightmodeler replace Modaic?
No. Modaic runs and calibrates the judgment calls your product makes, such as triage, moderation, or an LLM judge's verdict, and uses expert feedback to improve them. rightmodeler makes no production decisions. It is an offline audit that answers which model each step of your agent needs, and every swap it proposes waits for a human to review.
Does rightmodeler integrate with Modaic?
Not today. There is no --evaluator modaic and no Modaic code in the CLI. An evaluator is planned against Modaic's hosted API, so a judge you keep in your Modaic account would grade rightmodeler's replays; building it is waiting on Modaic confirming its canonical API. The integration page lists what exists today and what the plan will guarantee.
Why Modaic's hosted API and not its open-source SDK?
Because that is where Modaic's judging happens. Per its code, the open-source SDK requests Modaic's judge runs and confidence scores from Modaic's hosted service, and alignment runs only there; on its own, it is a DSPy program without those features. The trade is real: the integration will need a Modaic account and key, each graded case will go to Modaic's service, and Modaic's charges will sit outside rightmodeler's cost estimate and spend cap.
Can Modaic's confidence tell me a cheaper model is safe to swap in?
Not on its own. Modaic's own docs describe confidence as most useful as a ranking and thresholding signal, not a substitute for measuring agreement against labeled examples. rightmodeler's evidence is agreement with the outputs you accepted, with the sample size and any abstentions attached to each step.
Is Modaic open source?
Partly. As of 2026-09-22, its Python SDK is on GitHub under the MIT License with additional terms, which bar modifying it to change its default hub endpoint. The platform that serves decisions, confidence scores, and alignment is a hosted service, priced from $0 a month with a $10 usage credit on its entry plan.
Run the audit on your own traces
The CLI runs from npx, nothing to install, and your own traces settle the question.