Comparison · Artificial Analysis

rightmodeler vs Artificial Analysis

Optima, from Artificial Analysis, turns your tasks, files, or traces into a custom benchmark and ranks models on it by score, cost per task, and time per task. rightmodeler starts from the call sites in your code: it replays the recorded steps through cheaper or newer models and, when a swap clears every gate, opens a draft pull request you review.

Competitor · a leaderboard vs a pull requestVisit Artificial Analysis  (opens in a new tab)

TL;DR

Competitors on one question, answered in different units. Artificial Analysis, which calls itself the independent benchmarking company for AI, launched Optima on 2026-08-13. As of 2026-09-24, per its pages, you build a benchmark from your own files or a Hugging Face dataset, from traces imported from platforms such as Arize, Braintrust, and Langfuse, from a coding-agent session, or from a description of your use case, then run it across leading models and compare score, cost per task, and time per task. Its launch article says Optima lets you find an equally performant alternative to your current setup at 10x lower cost or time per task, which is the question rightmodeler asks too. The difference is where the answer lands. Optima ranks models on the tasks and trajectories you bring. rightmodeler, a free MIT-licensed CLI, ties recorded steps to the call sites in your code they came from, replays them there through cheaper or newer models from your provider's live catalog, judges each candidate against the output you already shipped, and turns each swap that clears every release gate into a draft pull request that changes only model identifiers, for a human to merge.

Optima: a benchmark built around your work

  • Ways in, per its pages: describe the work and attach a few examples, upload a dataset from your files or Hugging Face, connect Arize, Braintrust, Langfuse, LangSmith, or Opik to turn recorded traffic into tasks, or install the Optima skill in a coding agent to stage a benchmark from your repo, docs, or tickets.
  • Task styles for questions with known answers, questions about your uploaded files, agentic work that uses tools in a sandbox and hands in deliverables, and conversations with simulated users. Your own agent can compete in the same run over HTTP.
  • Grading by rubric criteria, or by a panel of judges that compares two responses side by side, calibrated on a few votes of your own and turned into an Elo rating: the pairwise approach Artificial Analysis uses for GDPval-AA and AA-Briefcase.
  • A leaderboard with score, cost per task, and time per task for each model, category-level results and custom metrics, which you can keep current as new models are released.

rightmodeler: the model behind each call site in your code

  • Starts from traces you export, including Langfuse, Braintrust, LangSmith, and OpenInference, and the repository that produced them. Each traced case is tied to the call site in your code it came from and replayed only there; a case that cannot be tied to a single call site is left out of the replay sample.
  • Candidates are models from your configured provider's live catalog that cost less than the one a step runs today.
  • Each candidate is judged against the output you already accepted for that exact step. Verdicts carry reference agreement, sample size, and abstentions, and a winner must clear the quality floor on held-out cases.
  • A swap that feeds later model steps must be confirmed end to end in a sandbox before a pull request opens, and a swap that clears every release gate becomes a draft pull request that changes only model identifiers; a human reviews and merges it.

Same question, different unit

Where each one starts, what it measures against, and what you get back.

artificial-analysis vs rightmodeler
question
Optima · the best model for your task, or an equally performant one at lower cost or time
rightmodeler · which call sites in your code can move to a cheaper or newer model
starting point
Optima · your files, a dataset, imported traces, a coding-agent session, or a description
rightmodeler · your recorded traces and the repository that made them
unit
Optima · a task in the benchmark; your whole agent can compete over HTTP
rightmodeler · one recorded step, bound to the call site it came from
models tried
Optima · the leading models you run the benchmark across
rightmodeler · cheaper models in your configured provider's live catalog
grading
Optima · rubric criteria, or a pairwise judge panel you calibrate
rightmodeler · agreement with the output you accepted, with abstentions and a held-out test against the quality floor
output
Optima · a leaderboard with cost and time per task
rightmodeler · a per-step report and, for each swap that clears every gate, a draft pull request
where it runs
Optima · hosted by Artificial Analysis; you sign in
rightmodeler · your machine, your provider key; no server, account, or telemetry

Pick by situation

Three situations, with the honest call in each.

You are building a contract-review agent that uses tools and hands in documents, nothing has shipped yet, and you need to pick a model.

the right hire: Artificial Analysis

With no production traffic there are no traces to replay, and rightmodeler replays plain-text steps only. Optima can start from a description and a few example contracts: its build agent drafts tasks and rubrics with you, agentic tasks run each model with tools in a sandbox and grade the deliverables, and every result shows cost and time per task.

Your agent has run in production on a frontier model for months, and you want each step that can hold a cheaper model changed in code, as a diff your team reviews.

the right hire: rightmodeler

A leaderboard ranks models on the benchmark; turning it into a change at each call site is a separate step. rightmodeler binds recorded steps to the call sites they came from, replays them there through cheaper or newer candidates, judges each candidate against what you shipped, abstains where the sample is too small, requires a winner to clear the quality floor on held-out cases, confirms swaps that feed later model steps end to end in a sandbox, and opens a draft pull request that changes only model identifiers.

A cheaper model scores near the top of your Optima benchmark, and your provider serves it.

the right hire: both, together

Optima tells you it does your kind of task well. Name its ID with --candidate and the CLI tests it at the call sites your traces came from, whatever its release date, as long as it costs less than the model a step runs today. One is evidence about the task; the other is evidence about each line of code that would change.

Test an Optima pick at each call site

There is no integration; the handoff is a model ID you copy.

Pass each model your Optima leaderboard ranked well with its own --candidate flag, spelled the way your provider's catalog spells it, and point the CLI at your recorded traces. The CLI tests every model you name and no others, whatever its release date, wherever it costs less than a step's current model, and each result comes back with its sample size and any abstentions.

The run happens on your machine with your own provider key. There is no rightmodeler server, no account with us, and no telemetry, and the CLI is never in your request path.

# each Optima pick gets its own --candidate

The honest part

  • Optima grades work rightmodeler does not replay. Its agentic tasks run models with tools in a sandbox and score the files they hand in, and its interaction tasks simulate whole conversations with different kinds of users. rightmodeler resends each recorded conversation as text: a conversation that carries tool definitions or tool calls is left out of the replay sample, and a call site that needs tools or structured output ends with no verdict.
  • Optima works before you have traffic. Describe the work and attach a few examples, and its build agent drafts the tasks and rubrics with you. rightmodeler needs recorded traces and has nothing to say about a step that has not run.
  • Optima looks across the field, so it can tell you when a stronger model is worth paying for. rightmodeler only tests models cheaper than what a step runs today, and only ones your configured provider lists.
  • Optima's grading handles tasks with no single right answer: a judge panel you calibrate compares two responses head to head. rightmodeler's reference is the output you shipped, so its verdicts measure agreement with that output, not ground-truth correctness.

Terms this page uses

custom benchmark
What Optima builds: tasks drawn from your files, a dataset, imported traces, a coding-agent session, or a description, with rubrics or pairwise grading attached, run across models under the same conditions.
pairwise judging
Optima's grading for tasks with no single right answer, per its page: a judge panel reads two models' responses side by side and picks the better one, and wins and losses become an Elo rating. You calibrate the panel with a few votes of your own first.
trace
The recorded log of an agent run: each model call with its inputs, outputs, and tool calls. Optima can turn imported traces into benchmark tasks; rightmodeler replays the recorded steps themselves, each at the call site it came from.
reference evidence
Agreement between a cheaper candidate's answer and the output you accepted for that exact step. Evidence of agreement with shipped output, not ground-truth correctness.
quality floor
The configurable minimum a candidate's worst-case agreement must clear on held-out cases, adjusted for the number of candidates, before rightmodeler recommends it.
abstain
What rightmodeler does when a step's sample is too small or its evidence incomplete: it recommends nothing rather than guessing.

Frequently asked questions

Is Optima a competitor to rightmodeler?

On one question, yes: both look for a cheaper model that holds up on your own work, and Artificial Analysis's launch article says Optima lets you find an equally performant alternative to your current setup at 10x lower cost or time per task. They answer it in different units. Optima ranks models on a benchmark you build; rightmodeler validates cheaper or newer models per call site in your code and hands you the change as a draft pull request. You can use Optima to shortlist and rightmodeler to validate each call site.

Can both start from the same traces?

Often, yes. Per its page, Optima connects to Arize, Braintrust, Langfuse, LangSmith, or Opik and turns recorded traffic into tasks. rightmodeler reads exported trace files from your own disk, including Langfuse, Braintrust, LangSmith, and OpenInference, the span format Arize Phoenix records. Optima builds benchmark tasks from them; rightmodeler replays the recorded steps at the call sites they came from.

Is there an integration between rightmodeler and Artificial Analysis?

No. rightmodeler has no Optima reader and no Optima evaluator; the external evaluators it ships are Braintrust, Langfuse, LangSmith, and promptfoo. The connection is manual: name a model your Optima leaderboard ranked well with --candidate.

Is this the Artificial Analysis whose scores The Open Frontier shows?

Yes. The Open Frontier's methodology page says every score on that site is Artificial Analysis's, taken from its public model pages and capability boards. This page is about Optima, Artificial Analysis's product for benchmarking models on your own tasks; for the public boards, see rightmodeler vs The Open Frontier.

Where does my data go?

Optima is a hosted product you sign in to. Per its page, you upload files, connect a trace platform, or upload the bundle the Optima skill stages in your coding agent, which sends nothing until you upload it. With rightmodeler, the CLI reads traces from your disk, and replays and judge calls go to the provider you configure, on your key; there is no rightmodeler server, account, or telemetry.

What does each one cost?

As of 2026-09-24, Optima's page says building and running a benchmark is charged at the raw token cost of the models used, with nothing added, and grading is priced per unit judged: $0.002 per rubric criterion per model with standard judges or $0.040 with premium judges, and $0.006 per pairwise match with standard judges or $0.150 with premium. The rightmodeler CLI is free and MIT-licensed; replays and judge calls are billed by your provider on your key, and rightmodeler estimate projects that spend before any paid call.

Start with the traces you already have.

The CLI runs from npx, nothing to install, and validates cheaper or newer models on your own traces.

View on GitHub