Comparison · promptfoo
rightmodeler vs promptfoo
promptfoo evaluates what you are about to ship against test cases you author. rightmodeler validates cheaper or newer models on what you already shipped, against the outputs you accepted, so the real line between them is where the test set comes from.
TL;DR
Not rivals for the same job. You hire promptfoo to test what you are about to ship: you author the prompts, cases, and assertions, and its CI gate fails the build when a pass rate drops. You hire rightmodeler to validate swaps to cheaper or newer models on what you already shipped: the CLI reruns the recorded calls in your traces on cheaper or newer candidates and scores every proposed swap against the output you accepted, step by step. Red teaming (generating adversarial attacks against your app) is entirely promptfoo's turf. promptfoo ships as a rightmodeler evaluator provider, selected with --evaluator promptfoo, pointed at your assertions file with --evaluator-config, and given each metric to grade with --evaluator-scorer; it is not a trace source or replay provider, and rightmodeler does not edit your promptfoo config.
Your test set is cases you author
- Nothing has shipped yet, or a prompt is changing and you need to show nothing regressed. For a system with no production history, authored cases are the only possible evidence.
- You want assertions you control, on quality, cost, and latency, run on every pull request, with a build that fails when the pass rate drops.
- You need adversarial cases: prompt injection, jailbreaks, PII leaks, access-control flaws. promptfoo generates these; rightmodeler does not. Start with promptfoo.
Your test set is traces you already have
- Your agent has served real users, and the outputs you accepted are the evidence you trust most.
- You want to know which steps can move to a cheaper or newer model without authoring a single case; the CLI treats your accepted output as the reference for that exact step.
- You want every step to carry its own verdict, how many samples backed it, and an abstention (rightmodeler recommends nothing for a step when the evidence is thin). Start with rightmodeler.
Benching models with promptfoo alone
- You name the exact model IDs in promptfooconfig.yaml and author the cases they compete on.
- The matrix view puts outputs side by side across prompts and models; you read it and decide.
- Coverage ends where your authoring did: production steps nobody wrote a case for go unmeasured.
With rightmodeler beside your suite
- Keep the suite. rightmodeler does not touch it, replace it, or sit in your request path.
- Cheaper candidates are shortlisted from the active provider's live catalog, not a hand-picked list, replayed against your real steps, and judged by a neutral third-family judge (a judge model from a family neither the current model nor the candidate belongs to) that grades both output orders, or by your own promptfoo assertions, deterministic types such as is-json and regex included.
- Each verdict carries reference agreement (how often the candidate matched the output you accepted), a quality floor (a configurable minimum that agreement must clear), sample size, and abstentions. A swap at a step that feeds later model steps counts only once it is confirmed end to end in a sandbox, so cascade risk (a cheaper step's output quietly breaking the step after it) surfaces before you swap.
- When you merge a swap, the new model ID goes into promptfooconfig.yaml by hand, and your CI gate keeps the decision honest from then on. promptfoo also ships as a rightmodeler evaluator provider, selected with --evaluator promptfoo, pointed at your assertions file with --evaluator-config, and given each metric to grade with --evaluator-scorer; it is not a trace source or replay provider.
Pick by situation
Three days at work, and the tool that honestly wins each one.
You rewrote your system prompt and need to know nothing regressed before Friday's deploy.
the right hire: promptfoo
Pre-ship regression on authored cases is promptfoo's core loop: run the suite, gate the merge in CI, fail the build if the pass rate drops. rightmodeler never evaluates prompts; it changes which model runs a step, never the prompt.
Security review is next month and you must show your support agent resists prompt injection and does not leak PII.
the right hire: promptfoo
Adversarial testing is promptfoo's headline: plugins generate attacks against your app as a black box, covering injection, jailbreaks, PII leaks, and authorization flaws, agents and RAG included. rightmodeler does no red teaming at all.
Your agent has run in production for months, one expensive model sits on every step, and you want to know which steps a cheaper model can actually hold.
the right hire: rightmodeler
There is no suite to write for this, and writing one would recreate what your traces already contain. The CLI turns the traces into the test set: the recorded calls rerun on cheaper or newer candidates, each candidate is scored against the answer you shipped, and every verdict arrives with its sample size and any abstentions. Where the evidence is thin it abstains rather than calling a winner, and the swaps that clear every gate arrive as a draft pull request for a person to merge.
What we will not pretend to do
- Red teaming and security testing belong to promptfoo, full stop. Plugin-generated attacks on agents and RAG apps, prompt injection, jailbreaks, PII leaks, authorization flaws: rightmodeler does none of this, and a validation run will never tell you your agent is secure.
- promptfoo works before an app has ever served a user. rightmodeler requires real traces, so it has nothing to say about a system with no production history.
- promptfoo can run on every pull request and fail the build. The rightmodeler CLI takes a different seat in CI: a scheduled job you choose to run, streaming JSON events with exit codes as verdicts, leaving a report behind; rightmodeler apply then opens a draft pull request for the swaps that clear every gate. Not observability. Not a runtime gateway. It runs offline on traces you already recorded, never in your request path.
- And rightmodeler's own limits, stated plainly: a passing score means the candidate agreed with what you shipped, not that either answer was correct; the production result is the reference, not ground truth. Thin evidence on a step ends in an abstention, not a guess, and some steps come back with no verdict at all.
Plain words for the terms above
- trace
- The record of an agent run: each step's inputs, outputs, and tool calls. rightmodeler's raw material; promptfoo does not need one.
- reference evidence
- Agreement between a candidate model's answer and the output you already accepted for that exact step. Evidence of agreement with shipped output, not ground-truth correctness.
- quality floor
- The configurable minimum agreement score a candidate must clear before rightmodeler will recommend it for a step.
- abstain
- What rightmodeler does when a step's sample is too small or its evidence incomplete: it recommends nothing rather than guessing.
- cascade risk
- The chance that a cheaper model's output at one step quietly breaks a later step. Swaps that feed later model steps are confirmed end to end in a sandbox so it surfaces.
Frequently asked questions
Does rightmodeler replace promptfoo?
No. promptfoo tests what you are about to ship against cases you author, and red-teams it for vulnerabilities. rightmodeler asks a different question of every step you have already shipped, and answers it with receipts: can a cheaper or newer model do this step's job? Keep your suite; the CLI runs offline on your own traces and is not a runtime gateway.
Is there an integration between rightmodeler and promptfoo?
promptfoo ships as a rightmodeler evaluator provider, selected with --evaluator promptfoo, pointed at your assertions file with --evaluator-config, and given each metric to grade with --evaluator-scorer, plus --evaluator-command when promptfoo is not on your PATH. It is not a trace source rightmodeler reads or a replay provider it runs through, so the handoff to your suite remains manual. Once you merge a swap, the new model ID goes into promptfooconfig.yaml by hand, and the validated step's accepted outputs can become assertions your CI gate enforces on every future change.
Can promptfoo compare models the way rightmodeler does?
Both can put models side by side, but the inputs differ. promptfoo compares the exact model IDs you name, on cases you author, in a matrix you read. rightmodeler shortlists cheaper candidates from the active provider's live catalog and uses your production traces as the test set, so nothing needs authoring, and every verdict carries reference agreement, sample size, and abstentions. If you have no traces yet, promptfoo is the only option of the two.
Is promptfoo still open source now that it is part of OpenAI?
Yes, as of this writing. promptfoo's own site and README state that promptfoo is part of OpenAI, the CLI remains MIT licensed and actively maintained, and OpenAI has publicly committed to keeping the open-source suite under its current license and to supporting existing customers. Its positioning today leads with AI security, with red teaming, guardrails, and code scanning alongside the original evals.
Does rightmodeler test or improve my prompts?
No. It is not a prompt-improvement or prompt-rewriting tool: it changes which model runs a step, never the prompt. Prompt regressions are exactly what your promptfoo suite is for.
Start with the traces you already have.
The CLI runs from npx, nothing to install, and validates cheaper or newer models on your own traces.