Comparison · Helicone
rightmodeler vs Helicone
Helicone sits in your request path, routing each call to a provider and logging what it cost and returned. rightmodeler starts from those logged requests and asks what the log cannot: would a cheaper model have produced a response you would have accepted?
TL;DR
Complements on the same traffic. You hire Helicone to run and read your model traffic: as of 2026-09-24, its docs describe an OpenAI-compatible AI Gateway in front of 100+ LLM providers that routes each request to the cheapest provider serving the model you named and fails over when one errors, with every request logged alongside its cost, latency, and errors, and sessions, custom properties, alerts, caching, and rate limits on top. You hire rightmodeler, a free MIT-licensed CLI, when the bill raises a different question: which of those calls could a cheaper model answer as well? Its dedicated Helicone adapter reads the request rows you export, replays each recorded call through cheaper candidates, judges them against the response you shipped, and reports agreement, sample size, and abstentions per step family. What clears every gate becomes a draft pull request that changes model identifiers only, for a human to review and merge. Keep Helicone in the request path; the audit never enters it.
How they differ
Helicone works on every live request. rightmodeler works once, on requests you already made.
Can you use both? Yes, and Helicone stays in the path
- Helicone keeps proxying production traffic throughout: no SDK change, no new header, no code in your app. The audit reads a file of request rows you exported, and nothing connects to your Helicone account while it runs; your Helicone key is used only by the export call you run yourself.
- The division of labor is simple. Helicone shows where the money goes, by session, user, feature, or any custom property you attach. The audit shows which of those calls a cheaper model could have answered as well, with evidence per step family. If you already stamp requests with Helicone-Session-Id, those rows arrive as one trajectory, so a multi-call agent run stays one unit in the statistics.
The supported workflow: Helicone rows in, a reviewed pull request out
A dedicated adapter reads Helicone request exports. Three commands, run from your repository.
Export your request rows with Helicone's query API, POST https://api.helicone.ai/v1/request/query, with includeInputs set so request_body and response_body come back with each row. The adapter recognizes Helicone rows by their request_id, response_id, and request_created_at keys, reads the model, messages, and any tool definitions from request_body, and takes response_body as the answer you shipped. Rows that share a Helicone-Session-Id become one trajectory; rows without one fall back to their request_id.
The preview run spends nothing. The full run replays each recorded call on cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env, judges every replay against the response you accepted, and caps replay and judge spend together at the figure you pass with --max-cost-usd. It writes report.md with a verdict per step family, and when a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers only. There is no rightmodeler server, no account, and no telemetry.
# export request rows, bodies included, to a file
# preview the pipeline without spending anything
# then run it against your replay endpoint
Which one gets the call?
Three situations, with the honest winner named in each.
Your provider starts returning errors in the middle of a launch, and requests need somewhere else to go right now.
the right hire: Helicone
That is a live request that needs saving, and it is Helicone's job: per its docs, the AI Gateway fails over to the next provider serving the same model on rate limits, timeouts, and server errors, and alerts can fire on error rate. rightmodeler is offline by design and cannot help a request in flight.
You already know which agent drives the bill, and every step in it calls the same frontier model.
the right hire: rightmodeler
Helicone's cheapest-provider routing lowers the price of the model you named; it never changes the model. The audit replays that agent's recorded calls on cheaper models from your provider's live catalog, judges each against the response you shipped, abstains where the sample is thin, and re-clears the quality floor on held-out cases before it recommends a swap.
Finance asks why the model bill doubled last month and what to do about it.
the right hire: both, together
Helicone answers the first half: sessions and custom properties show which workflow, user, or feature drove the spend, per its docs. Export those rows and the audit answers the second half, per step family: which calls a cheaper model could have answered as well, with the evidence attached. An approved swap keeps flowing through Helicone like any other request.
The honest part
- Helicone runs in your request path and keeps it up: routing across providers, failover, response caching on Cloudflare's edge, and custom rate limits by request count, cost, or property, per its docs. rightmodeler never serves a single production request and cannot retry, cache, or rate-limit anything.
- Helicone is where you see the traffic: a searchable log of every request, sessions that lay out an agent run, user and custom-property analytics, alerts on errors, cost, and latency, and weekly reports to email or Slack. rightmodeler is a point-in-time report you choose to run on rows you exported.
- Helicone is open source under the Apache License 2.0 and can be self-hosted with Docker Compose or Kubernetes, per its docs. Its gateway also gives one key access to many providers with pass-through billing. rightmodeler brings no models and no keys of its own; it replays through the endpoint and key you configure.
- The audit's verdicts measure agreement with the response you shipped, which is evidence of agreement, not proof of correctness, and a family with thin evidence abstains instead of recommending.
Terms this page uses
- session
- In Helicone, the related requests grouped by the Helicone-Session-Id header, per its docs. The audit reads that ID as the trajectory, so one agent run counts as one unit.
- trace
- The step-by-step record of what your agent did. For Helicone users it is the request log: each exported row holds the request body, the response body, and the tokens used.
- swap candidate
- A cheaper model from your configured provider's live catalog, replayed on one step family's recorded requests.
- reference evidence
- Agreement between a candidate's response and the response you shipped for that exact request. Evidence of agreement with shipped output, not proof of correctness.
- quality floor
- The configurable minimum a candidate must clear, on the cases it was chosen on and again on held-out cases, before the audit recommends it.
- abstain
- What the audit does when a step family's evidence is too thin: it recommends nothing and names the reason. The current model stays.
Frequently asked questions
Does rightmodeler replace Helicone?
No. Helicone proxies, caches, and observes live traffic. rightmodeler runs offline on rows you exported and answers a different question: whether a cheaper model would have produced an equivalent response on the calls you actually made. Keep Helicone in the request path; an approved swap keeps flowing through it.
Helicone's gateway already routes to the cheapest provider. Isn't that the same saving?
It is a different saving. Per Helicone's docs, provider routing finds every provider that serves the model you requested and sends the call to the cheapest available one, so the model stays the one you named. The audit tests a different model for each step, judges it against the response you shipped, and recommends it only when it holds up. The two stack: a cheaper model, served by the cheapest provider.
Does the audit use Helicone's recorded cost?
Not as evidence. The audit prices candidate calls from what your replay endpoint reports for the call it just ran, and marks the figure as an estimate when the provider returns no cost. Historical spend stays in Helicone, where you already track it.
What does Helicone joining Mintlify mean for this workflow?
Helicone announced on 2026-03-03 that it had been acquired by Mintlify, and as of 2026-09-24 its post says Helicone's services remain live in maintenance mode, with security updates, new models, and bug and performance fixes still shipping. The audit reads request rows from a file on your disk and never connects to Helicone while it runs, so it works on any export you already have.
Does my request data leave my machine?
Only during replay, when recorded inputs go to candidate models through the endpoint you configured, under your own key. There is no rightmodeler server and no account; the export file and the report stay on your disk.
Will it always find savings?
No, and that is deliberate. A family abstains when the evidence is thin, when the excluded fraction runs over the ceiling, or when no candidate clears the quality floor on held-out cases. An audit that always finds a swap is not measuring anything.
Run the audit on your own traces
The CLI runs from npx, nothing to install, and your own traces settle the question.