Comparison · DeepEval
rightmodeler vs DeepEval
DeepEval, with Confident AI as its platform, gives you the parts of an evaluation practice: metrics, pytest-style tests, synthetic datasets, and production monitoring. rightmodeler runs one narrower workflow on the traces you already recorded: which steps can move to a cheaper model, with per-step evidence and a draft pull request.
TL;DR
Complement, with a clean handoff. As of 2026-09-24, DeepEval describes itself as an open-source LLM evaluation framework: pytest-style unit tests for LLM outputs, 50+ ready-to-use metrics per its docs, synthetic datasets, and end-to-end, trajectory, and component-level evals that run in your own environment. Confident AI, the company that builds DeepEval, sells the platform around it: shared test reports, regression tracking across runs, tracing with online evals, dataset management, red teaming, and governance. rightmodeler, a free MIT-licensed CLI, does one narrower job: it reads the traces your agent already recorded, shortlists cheaper models from your provider's live catalog, replays each step, judges every candidate against the output you accepted, and opens a draft pull request that changes model identifiers only. rightmodeler has no dedicated integration with either product. The handoff is your CI: the draft pull request goes through your deepeval test run job, or Confident AI's PR Eval Gate, like any other change before a person merges it.
DeepEval and Confident AI: one team, two layers
- They are not the same product, and they come from the same company. DeepEval's own FAQ puts it plainly: DeepEval is the open-source evaluation framework, while Confident AI is the enterprise platform built for teams that need managed evals, collaboration, observability, and production workflows. deepeval.com names Confident AI Inc. as its organization, and Confident AI says it builds and maintains DeepEval and DeepTeam, its open-source red teaming framework. Per DeepEval's docs, you can use either one without the other.
- DeepEval, Apache-2.0 on GitHub, is the layer that runs where your code runs. You attach metrics to test cases, from LLM-as-a-judge metrics such as G-Eval to graph-based DAG metrics and non-LLM checks such as exact match and JSON correctness. You assert on them in pytest-style tests that deepeval test run executes in CI, exiting non-zero on failure. When real data runs short, it synthesizes goldens from your documents and simulates conversations. It needs no account, only a model provider key for metrics that use an LLM judge.
- Confident AI is the layer your team shares. It keeps datasets and test runs, compares regressions across runs, and posts a pass, fail, or neutral GitHub check on every pull request through its PR Eval Gate. It traces production with online evals on traces, spans, and threads, detects drift, and alerts. Its 2026-09-11 changelog introduced confident-trace, an open-source, OpenTelemetry-first tracing SDK that it now recommends for production tracing, while DeepEval stays the evaluation framework. As of 2026-09-24, its pricing page lists a free plan, paid plans from $200 a month, and self-hosting on Enterprise plans.
How they differ
Both can tell you whether a different model holds up. What each answer is built from is different.
Can you use both? Yes, and the handoff is your CI
- Nothing about your DeepEval setup changes. Your tests, metrics, and goldens stay where they are, Confident AI, if you use it, keeps tracing and scoring production, and rightmodeler never sits in the request path or asks for new instrumentation. There is no dedicated integration between them, and the workflow below needs none.
- The two meet in two places. On the way in, rightmodeler reads traces from files, and an OpenTelemetry pipeline that already feeds Confident AI can write those files too. On the way out, rightmodeler's change is an ordinary draft pull request, so whatever your repository runs on pull requests, DeepEval tests or Confident AI's PR Eval Gate, runs on it before anyone merges.
The handoff, step by step
Four commands, run from your repository. Nothing connects to Confident AI; your DeepEval checks run where they already run.
Traces in. Confident AI's trace-broadcasting docs show an OpenTelemetry Collector sending every span to Confident AI and to a second destination. Make that second destination a file exporter and the spans land on disk as OTLP JSON. rightmodeler reads the file when the spans carry OpenTelemetry GenAI attributes with the messages captured, or OpenInference attributes; Confident AI's OTEL server ingests both conventions natively, per its docs.
The audit. The preview spends nothing. The full run shortlists cheaper models from your provider's live catalog, replays each step family through the OpenAI-compatible endpoint you name with --base-url and --api-key-env, on your own key, judges every replay against the output you accepted, and writes report.md with a verdict or a named abstention per step family.
Through your checks. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers only and carries an evidence table. A workflow that runs deepeval test run on pull requests, the setup DeepEval's CI/CD docs show, holds the swap to your suite; Confident AI's PR Eval Gate posts its check run the same way. rightmodeler watch comments once when a check fails and closes the pull request if a check with the same name fails again under a new run on the same head commit. A person reviews and merges; the CLI never does.
# preview the pipeline against the spans your collector wrote, without spending
# replay cheaper candidates through your endpoint, capped at 25 USD
# open the model-only draft pull request your DeepEval checks run on
# on a schedule: answer reviews and react to failing checks
Which one does the job?
Three situations, with the honest winner named in each.
Your RAG assistant has not launched, you have a folder of product documents and no user traffic, and every prompt change has to prove it did not hurt faithfulness or relevancy before it merges.
the right hire: DeepEval
That is DeepEval's core loop, per its docs: synthesize goldens from your documents, attach metrics such as Faithfulness and Answer Relevancy, and let deepeval test run fail the build on a regression. rightmodeler needs recorded traces, and it changes which model a step calls, never the prompt.
Your DeepEval suite is green, your agent has run in production for months, and every step still calls the frontier model you launched with. Nobody can say which steps need it.
the right hire: rightmodeler
A green suite says the agent clears your bar; it does not try cheaper models step by step. rightmodeler shortlists them from your provider's live catalog, replays every step family on the traces you recorded, judges each candidate against the output you accepted, and reports sample size and abstentions, then opens a pull request only for the swaps that clear every gate.
Your team's rule is that nothing merges without passing the DeepEval suite, and you want the model bill down without breaking that rule.
the right hire: both, together
rightmodeler proposes the swaps with evidence attached, as a draft pull request. Your deepeval test run job, or Confident AI's PR Eval Gate, checks it like any other change, and rightmodeler watch closes the pull request if the same check fails again on a new run of the same commit. The rule holds and the evidence comes to the reviewer.
The honest part
- DeepEval is a whole evaluation practice: metrics for agents, RAG, multi-turn conversations, safety, images, and voice; pytest-style tests for CI; synthetic goldens and simulated conversations; prompt optimization; and standard benchmarks. rightmodeler ships none of that. It answers one question and expects your suite to remain the bar.
- DeepEval works before launch. It can build goldens from your documents when no traffic exists, while rightmodeler needs recorded traces and has nothing to say about an agent that has not run.
- Confident AI watches production: online evals on every trace, span, and thread, drift detection, alerting, red teaming, and governance, with self-hosting on Enterprise plans. rightmodeler works after the fact on exported files and has no live view, by design.
- rightmodeler replays plain-text steps. A recorded case whose conversation carries tool calls, tool definitions, or non-text parts is left out of the replay sample, and a call site that needs tools or structured output is not replayed. DeepEval's agentic metrics, such as Tool Correctness and Argument Correctness, score exactly those steps.
- And the audit's verdicts say only what they measure: agreement with the output you shipped, which is evidence, not proof of correctness. Where the evidence runs thin, the audit abstains instead of recommending.
The words this page turns on
- golden
- DeepEval's term for a dataset entry: an input or scenario, plus the expected output or outcome when you have one. At evaluation time your app runs on it and it becomes a test case. You write goldens, pull them from Confident AI, or synthesize them.
- trace
- The record of what your agent did: every model call with its inputs and outputs. rightmodeler reads traces from exported files, such as the OTLP spans an OpenTelemetry Collector writes to disk.
- LLM-as-a-judge
- Using a model to grade outputs. DeepEval's G-Eval applies it to criteria you write. rightmodeler picks its judge from a model family neither the candidate nor the current model belongs to, so nothing grades its own work.
- reference evidence
- Scores measured against the output you actually accepted for that exact step. Evidence of agreement with shipped output, not proof of correctness: the production result is the reference, not ground truth.
- quality floor
- The minimum agreement a candidate must clear, and clear again on held-out cases, before the audit recommends a swap. It is configurable, and anything below it is a no.
- abstain
- The audit's decision to recommend nothing for a step family when the sample is small or the evidence is mixed, with the reason named. The current model stays.
Frequently asked questions
Is DeepEval the same as Confident AI?
No, per DeepEval's own FAQ: DeepEval is the open-source evaluation framework, and Confident AI is the enterprise platform for teams that need managed evals, collaboration, observability, and production workflows. They come from the same company, Confident AI Inc., and DeepEval can send its results to Confident AI when you add a Confident AI key. This page covers both because they form one team's stack.
Can't DeepEval compare models already?
It can, per its docs. You rerun the same goldens on another model version, label each test run with the model as a hyperparameter, compare the runs, and Arena G-Eval can pick a winner between versions. You choose the model to try and build the goldens first. rightmodeler starts from the traces you recorded, shortlists the cheaper candidates itself, and returns a verdict per step family with sample size and abstentions, ending in a model-only draft pull request.
Can DeepEval's metrics grade rightmodeler's replays?
The CLI's external evaluators are Braintrust, Langfuse, LangSmith, and promptfoo, and DeepEval is not among them. Without one, the built-in judge grades each replay: a model from a family neither the candidate nor the current model belongs to, called twice with the positions swapped. Your DeepEval metrics grade the swap where they already run: in CI, on the draft pull request, before anyone merges.
Will my DeepEval checks run on rightmodeler's pull request?
Yes, whenever your workflows run on pull requests, because the swap arrives as an ordinary draft pull request. One GitHub rule applies: when GitHub Actions' GITHUB_TOKEN opens the pull request, its pull_request runs wait until a person with write access approves them, and a GitHub App installation token avoids that, per rightmodeler's GitHub guide. For rightmodeler watch to see check runs, such as the one Confident AI's PR Eval Gate posts, use an App installation token or a classic token with the repo scope.
Does rightmodeler replace DeepEval or Confident AI?
No. DeepEval and Confident AI test, monitor, and govern your agent, and your team works in them every day. rightmodeler is an offline workflow for one decision: which model each step should call. Not observability. Not a runtime gateway. Its output is a pull request your existing checks review.
Is DeepEval open source, and what does Confident AI cost?
DeepEval is Apache-2.0 on GitHub and runs locally without an account. As of 2026-09-24, Confident AI's pricing page lists a free plan limited to two seats, one project, and five test runs a week, a Starter plan at $200 a month, a Team plan at $2,000 a month, and custom Enterprise pricing, with self-hosting on Enterprise plans. The rightmodeler CLI is free and MIT-licensed; replays bill to your own provider key.
Run the audit on your own traces
The CLI runs from npx, nothing to install, and your own traces settle the question.