Integration · the OpenAI SDK
rightmodeler + the OpenAI SDK
No tracing platform required. rightmodeler reads the request and response log your code already writes around the OpenAI SDK, measures cheaper candidates against accepted outputs, and reports the result call by call.
TL;DR
rightmodeler reads per-line OpenAI SDK request and response dumps with a dedicated adapter: each logged call becomes a step, with no tracing platform in between. Lines are recognized by a messages array beside a response object carrying choices, grouped into trajectories by case_id and ordered by timestamp. Successful steps are replayed against cheaper candidates through the OpenAI-compatible endpoint you configure and scored against the answer you already shipped.
How it works with the OpenAI SDK
Dump requests and responses as JSONL
Each line holds one call: a case_id naming the run it belongs to, the request's model and messages, and the response object with its choices and usage, the shape you get from dumping your chat.completions kwargs alongside response.model_dump(). If your agent already writes such a log, adding case_id is the only change. Save it anywhere on disk.
Autodetected, flat, one call per step
The adapter recognizes the format from a messages array next to a response object with choices, and requires case_id on every line. Lines sharing a case_id become one trajectory in timestamp order, the system messages are split out into the step's system prompt, the rest become its messages, and choices[0].message becomes the output. usage.prompt_tokens and usage.completion_tokens supply usage from the line or the response, whichever carries them.
Cheaper candidates, judged against what you shipped
Each step is replayed against cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env. A configured external evaluator scores the outputs when it is reachable; otherwise the built-in judge runs two temperature-zero calls with the reference and candidate positions swapped, choosing a judge from a family that is neither the candidate's nor the reference's. Verdicts are confidence lower bounds kept separate per evaluator kind.
Coupled steps re-run in a container
A step whose output can feed later model-authored steps is not judged from the log alone. rightmodeler runs your pipeline inside a container image you name in a --modeb-config file, mounting your app read-only and mapping each scanner step to the runtime header your app emits. Every model call leaves through a metered proxy that swaps the model per step and fails closed on a request with no correlation header, so the candidate does the real work and the spend is accounted for.
A per-step report, then a pull request
The run writes report.md and report.json with a verdict per step family and streams JSONL events as it works. Thin evidence produces an abstention with a named reason. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else, watch reconciles it against CI, and rollback restores the pre-apply files, refusing unless every restored digest matches.
Setup
Nothing to install: npx runs the published CLI. Hand it your request log. Nothing talks to your OpenAI account; the audit runs on the JSONL files already on your disk.
# sanity-check the shape: case_id, messages, response on each line
# preview the pipeline against the log, without spending
# then run it against your replay endpoint
# or let your coding agent drive it
What rightmodeler reads from the OpenAI SDK
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: case_id · messages · response.choices
Use cases
Audit an agent with no tracing platform
You do not need LangSmith, Langfuse, or an OTel collector to get a model audit. A plain JSONL log of your OpenAI SDK calls is enough for the full per-step treatment: replay, score, verdict, report, pull request.
Measure a candidate before you edit a model string
In OpenAI SDK code the model is a hardcoded string, and changing it is an act of faith. Replay last week's log first and read the verdict per step family, judged against outputs your users already accepted, before the string changes.
Bring the cost you already recorded
If your wrapper logs cost_usd and duration_ms per call, the adapter reads them, so the report compares candidate spend against a figure you measured rather than one it inferred. Lines without them are still replayed; only the historical side stays unknown.
The honest part
- There is no export button for this format. The SDK does not write these logs by itself; your code does, by dumping each request and response as a line of JSON with a case_id attached. If yours does not yet, that wrapper comes before the audit.
- Every line needs case_id, a messages array, and a response object with a non-empty choices array. A line missing any of them is dropped with a recorded reason rather than guessed at.
- Declared tools are not read from the request. The recorded answer, tool calls included, is what the candidate is scored against, so a step whose difficulty lived in the tool schema is judged on its result rather than its options.
- Evidence minimums are per evaluator kind: at least 10 assessed executions, at least two distinct step IDs, and at least five distinct trajectories. Below any of those the family abstains with a named reason instead of recommending.
- Confirmation for coupled steps runs your pipeline in a container and needs a --modeb-config file naming the image and the step map. Without it the recommendation is reported as unconfirmed rather than quietly promoted.
- It is a report, not a runtime gateway. rightmodeler never sits between your SDK and OpenAI, and nothing changes in production until you merge the pull request.
Frequently asked questions
What does each line of the log need to contain?
A case_id, a messages array, and a response object whose choices array is non-empty. model can sit on the line or on the response, usage on either, and timestamp orders the steps within a case. One JSON object per line, the request and its response together.
I do not log my OpenAI calls today. What is the minimum?
A thin wrapper around chat.completions.create that writes one JSON line per call: a case_id for the run, the request kwargs, and response.model_dump(). Most agent codebases already have a choke point every call passes through; that is the place.
Why does it need case_id?
Because the statistics cluster by trajectory. Cases from the same run are resampled together rather than counted as independent trials, so a ten-call agent run over five cases is not read as fifty. Without a case_id the adapter has nothing honest to cluster on, so it refuses the line.
Does rightmodeler replace my observability setup?
No. It is an offline audit that measures cheaper candidates against the outputs you accepted and reports the evidence for each call. If you later adopt LangSmith, Langfuse, or OTel GenAI, rightmodeler reads those traces too, same CLI, different adapter.
My code hardcodes the OpenAI client. How can it confirm end to end?
By running your pipeline in a container rather than patching your code. You name an image and a command in a --modeb-config file, your app is mounted read-only, and model calls leave through a metered proxy that swaps the model per step. Requests without a correlation header are refused rather than attributed by guesswork.
Will it always find savings?
No, and that is the point. A family abstains when the evidence is thin, when the excluded fraction runs over the ceiling, or when no candidate clears the quality floor again on held-out cases. A tool that always finds a swap is not measuring anything.
Run the audit on your own traces
The CLI runs from npx, nothing to install; your the OpenAI SDK traces do the rest.