Integration · the OpenAI SDK
rightmodeler + the OpenAI SDK
No tracing platform required. rightmodeler reads the request/response log your code already writes around the OpenAI SDK and proves, call by call, which ones can move to a cheaper model.
TL;DR
rightmodeler reads per-line OpenAI SDK request+response dumps with a dedicated adapter: each logged call becomes a step, with no tracing platform in between. Successful steps are replayed through cheaper candidates on OpenRouter and judged against the output you already shipped, with a quality floor of 0.90 by default; tool-using steps are confirmed by re-running your pipeline with the SDK redirected to the candidate.
How it works with the OpenAI SDK
Dump requests and responses as JSONL
Each line holds one call: the request's model, messages, and tools next to the response and its usage, the shape you get from dumping your chat.completions kwargs with response.model_dump(). If your agent already writes such a log, that file is the trace. Save it anywhere on disk, one file or a folder of them.
Autodetected, flat, one call per step
The ingester recognizes the format by its top-level model, messages, and usage keys and routes it to a dedicated adapter. Each line becomes one step in a flat sequence (no run tree to walk) with the system prompt split out, tool calls read from choices[0].message, and their arguments parsed even when they arrive as JSON strings.
Cheaper candidates, judged against what you shipped
Single-shot steps are replayed through cheaper OpenRouter models on the exact messages from your log, and a judge from a different model family scores each candidate against the output you already shipped, with a quality floor of 0.90 by default. JSONL logs carry no cost field, so the baseline is derived from the model name and token usage via OpenRouter pricing.
Tool-using steps re-run through your real code
Steps with tool calls or loops are confirmed end-to-end: the skill re-runs your pipeline in a throwaway git worktree, injecting a sitecustomize shim that monkeypatches openai.OpenAI and AsyncOpenAI (and langchain_openai.ChatOpenAI) to route to the candidate model on OpenRouter. Your code runs unmodified.
A per-step report you approve
The report lists each proposed swap with its evidence and confidence, and the TUI walks you through approve, reject, or hold, per step, never all-or-nothing. Weak evidence means an abstention, and high-risk steps stay on the current model.
Setup
Install once, then hand the skill your request log. Nothing talks to your OpenAI account; the audit runs on the JSONL files already on your disk.
# install the rightmodeler skill
# sanity-check the shape: model, messages, usage on each line
# inside Claude Code, hand the log to the skill
What rightmodeler reads from the OpenAI SDK
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: model · messages · usage
Use cases
Audit an agent with no tracing platform
You don't need LangSmith, Langfuse, or an OTel collector to get a model audit. A plain JSONL log of your OpenAI SDK calls is enough for the full per-step treatment: replay, judge, verdict, report.
Prove a swap before you edit a model string
In OpenAI SDK code the model is a hardcoded string, and changing it is an act of faith. Replay last week's log first and read the verdict per call (judged against outputs your users already accepted) before the string changes.
Confirm tool loops without touching your code
Because your pipeline speaks the OpenAI SDK, end-to-end confirmation can redirect it from the outside with a sitecustomize shim, not a code edit. Cascade effects surface on your real pipeline, unmodified, before you commit to a swap.
The honest part
- There is no export button for this format. The SDK does not write these logs by itself; your code does, by dumping each request and response as a line of JSON. If yours doesn't yet, that wrapper comes before the audit.
- JSONL logs carry no cost field, so the baseline is derived from the model name and token usage via OpenRouter's pricing table: good enough to rank swaps, not an invoice.
- Steps with tool calls or loops are not judged from the log alone; they are confirmed by re-running your pipeline end-to-end with the SDK redirected to the candidate model.
- High-risk step families (auth, payments, migrations, deploys, secrets) are flagged and left on the current model, and weak evidence means abstain rather than a recommendation.
- It is a report, not a runtime gateway. rightmodeler never sits between your SDK and OpenAI, and nothing changes in production until you change it.
Frequently asked questions
What does each line of the log need to contain?
Top-level model, messages, and usage are what the autodetector looks for; the response object with its choices supplies the shipped output, and tools is read when present. One JSON object per line, the request and its response together.
I don't log my OpenAI calls today. What's the minimum?
A thin wrapper around chat.completions.create that writes one JSON line per call: the request kwargs plus response.model_dump(). Most agent codebases already have a choke point every call passes through; that's the place.
Does rightmodeler replace my observability setup?
No. It is an offline audit that answers one question your logs don't: which calls can move to a cheaper model without losing quality. If you later adopt LangSmith, Langfuse, or OTel GenAI, rightmodeler reads those traces too, same skill, different adapter.
My code hardcodes the OpenAI client. How can it replay end-to-end?
The skill re-runs your pipeline in a throwaway git worktree with a sitecustomize shim on PYTHONPATH that monkeypatches openai.OpenAI and AsyncOpenAI to route to the candidate model on OpenRouter, using your own key. Your code runs unmodified, and the worktree is discarded afterward.
Does my log data leave my machine?
Replays send step inputs to candidate models through OpenRouter with your own API key, requests of the same shape your agent already sends to OpenAI. There is no rightmodeler server and no account; the report lands on your disk.
Will it always find savings?
No, and that is the point. Where the evidence is weak the audit abstains rather than recommends, and high-risk families (auth, payments, migrations) are flagged and left alone. A tool that always finds a swap is not measuring anything.
Run the audit on your own traces
The skill is free on GitHub. One command installs it; your the OpenAI SDK traces do the rest.