Integration · Codex CLI
rightmodeler + Codex CLI
Codex CLI already writes every session to disk as a rollout log. rightmodeler reads those files where they sit, measures cheaper candidates against accepted outputs, and reports the result turn by turn.
TL;DR
rightmodeler ingests Codex CLI rollout logs straight from ~/.codex/sessions with a dedicated adapter, no export step, because the JSONL files are already on your machine. The adapter keys on the session_meta record, takes the model from each turn_context, gathers that turn's user messages and assistant output from the response_item records, and derives per-turn usage from the token_count events. Each turn is then replayed against cheaper candidates and scored against the answer Codex gave.
How it works with Codex CLI
Your sessions are already on disk
Codex CLI writes each session to ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl as you work. There is nothing to export and nothing to instrument; choose one rollout or a day's folder.
Autodetected, then folded turn by turn
The adapter recognizes a rollout from its session_meta record (id, cwd, cli_version, model_provider) alongside turn_context, response_item, and event_msg lines. The session id becomes the trace, each turn_context opens a turn with its own model and turn_id, user messages become that turn's inputs, and assistant messages, function calls, tool-call outputs, and search calls become its output. Reasoning items and world-state metadata are skipped.
Cheaper candidates, judged against what you accepted
Each turn is replayed against cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env. A configured external evaluator scores the outputs when it is reachable; otherwise the built-in judge runs two temperature-zero calls with the reference and candidate positions swapped, choosing a judge from a family that is neither the candidate's nor the reference's. Verdicts are confidence lower bounds kept separate per evaluator kind.
A report, and the call stays yours
The run writes report.md and report.json with a verdict per step family and streams JSONL events as it works. Thin evidence produces an abstention with a named reason. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else, watch reconciles it against CI, and rollback restores the pre-apply files, refusing unless every restored digest matches.
Setup
Nothing to install: npx runs the published CLI, and init finds your Codex rollouts on disk automatically. Nothing connects to your OpenAI account; the audit runs on files Codex already wrote, against whichever replay endpoint you configure.
# list your most recent Codex rollouts
# preview the pipeline against a rollout, without spending
# then run it against your replay endpoint
# or let your coding agent drive it
What rightmodeler reads from Codex CLI
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: session_meta · turn_context · response_item
Use cases
Audit a stretch of coding sessions
Rollouts accumulate under one dated tree, so the audit can take a day or a week in a pass and show which step families genuinely earned the model they ran on, across everything you did.
Evidence before you change the default model
Instead of flipping your Codex model config and watching for regressions, replay the sessions you already ran and read the per-family verdicts first. The decision gets made on evidence from your own work, not a leaderboard.
Per-turn usage, not a running total
Codex reports cumulative token counts. The adapter subtracts each turn's baseline from the running total, so a turn's usage is its own rather than the session's so far, which keeps the cost side of the comparison honest.
The honest part
- It reads rollout files from disk. It does not hook into Codex, watch live sessions, or change your Codex configuration; acting on the report stays your move.
- A turn needs a turn_context with a model, and a response item needs a turn to attach to. Items that arrive before any turn context, or with an unrecognized payload type, are dropped with a recorded reason rather than guessed at.
- Turns that produced no assistant output are left out entirely, so a session of interrupted turns can yield fewer steps than it appears to contain.
- Codex does not log dollar cost. Historical spend is not derived from the rollout; candidate calls are priced from what your replay endpoint reports and marked as an estimate when the provider returns none.
- Evidence minimums are per evaluator kind: at least 10 assessed executions, at least two distinct step IDs, and at least five distinct trajectories. Below any of those the family abstains with a named reason instead of recommending.
- It is a report, not a gateway. Nothing sits between Codex and its provider, and there is no rightmodeler server to see your sessions.
Frequently asked questions
Where does Codex CLI keep its session logs?
In ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl, one JSONL file per session, written as you work. Pass a rollout or a day's folder with --traces; the format is autodetected from the session_meta record and the turn_context and response_item lines that follow it.
Do I need to export anything before running the audit?
No. Codex already writes its rollouts to local disk, so there is no dashboard to visit and no export API to call. The only credential the audit needs is the key for whichever OpenAI-compatible endpoint you replay through.
Does rightmodeler change which model Codex uses?
Not on its own. It produces per-family verdicts with the evidence attached, and where a family clears every gate it opens a draft pull request against your repository that you review and merge. Nothing hooks into Codex or intercepts its traffic.
How does it count tokens for a turn?
From the token_count events. Codex reports a running total, so the adapter takes the total at the end of a turn and subtracts the total carried in from the previous one, falling back to the last-call usage when no total is present.
Does my session data leave my machine?
Only during replay, when step inputs go to candidate models through the endpoint you configured, under your own key, the same class of call Codex made to produce the log in the first place. There is no rightmodeler server and no account.
What about turns where Codex ran commands or edited files?
Those tool calls and their outputs are part of the turn's recorded output, so the judge sees what the model actually did rather than a stripped answer. Where a turn's result can feed later model-authored steps, confirmation runs your pipeline in a container with the candidate swapped in, and without a --modeb-config the recommendation stays unconfirmed.
Run the audit on your own traces
The CLI runs from npx, nothing to install; your Codex CLI traces do the rest.