Source: https://www.rightmodeler.com/integrations/claude-code

# rightmodeler + Claude Code

rightmodeler runs inside the tool that wrote the traces. Claude Code already keeps every session on disk, so the audit starts with no export step, no SDK, and no re-instrumentation.

Claude Code writes every session to ~/.claude/projects as JSONL, so there is no export step at all. A dedicated adapter recognizes the transcript by its type, sessionId, uuid, parentUuid, and message keys, joins the rows of one assistant message back together, walks parentUuid up to collect the user turns that prompted it, and folds each assistant message into a step with its own token usage. Those steps are then replayed against cheaper candidates and scored against the answer you accepted.

## How it works with Claude Code

### 01 · Locate: Your traces are already on disk

Claude Code writes every session to ~/.claude/projects/<encoded-path>/*.jsonl as it runs, one folder per project, one JSONL file per session. There is nothing to export and no SDK to add; the audit starts from files you already have.

### 02 · Ingest: Each assistant message becomes a step

The adapter recognizes the transcript from user and assistant rows carrying sessionId, uuid, parentUuid, and a message object. Rows that share a session and a message id are stitched into one assistant message, its content blocks concatenated and its final usage kept. Walking parentUuid up to the previous assistant collects the user turns that prompted it, and any tool_result blocks from those turns ride along with the output. Metadata rows, from started and result to permission-mode and pr-link, are skipped.

### 03 · Replay + judge: Cheaper candidates, judged against what you accepted

Each step is replayed against cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env. A configured external evaluator scores the outputs when it is reachable; otherwise the built-in judge runs two temperature-zero calls with the reference and candidate positions swapped, picking a judge from a family that is neither the candidate's nor the reference's. Verdicts are confidence lower bounds kept separate per evaluator kind.

### 04 · Report + apply: A per-step report, then a pull request

The run writes report.md and report.json with a verdict per step family and streams JSONL events as it works. Thin evidence produces an abstention with a named reason. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else, watch reconciles it against CI, and rollback restores the pre-apply files, refusing unless every restored digest matches.

## Setup

Nothing to install: npx runs the published CLI, and init finds your Claude Code sessions on disk automatically. The coding-agent skill also runs inside the tool that wrote the traces. Sessions live at ~/.claude/projects; point the CLI at one project folder or a single session file.

```bash
npx rightmodeler init
```

```bash
# # your sessions are already on disk, one folder per project
ls ~/.claude/projects
```

```bash
# # preview the pipeline against a session, without spending
npx rightmodeler init --plan --traces ~/.claude/projects/<project>/<session>.jsonl --repo .
```

```bash
# # then run it against your replay endpoint
npx rightmodeler init --traces ~/.claude/projects/<project>/<session>.jsonl --base-url $BASE_URL --api-key-env PROVIDER_API_KEY --max-cost-usd 25
```

## What rightmodeler reads from Claude Code

| Normalized field | Source field |
| --- | --- |
| trace | sessionId |
| step grouping | sessionId + message.id |
| model | message.model |
| input messages | ancestor user turns via parentUuid |
| output | assistant content blocks |
| tool results | tool_result blocks in those turns |
| trajectory | promptId of the nearest user turn |
| token usage | message.usage.input/output_tokens |

## Autodetection

Detected by: `type · sessionId · parentUuid`

## Use cases

### Find out which turns earn the frontier model

Every assistant message carries its model and token usage. The audit replays those messages against cheaper candidates and reports, per step family, where the big model is earning its price and where a smaller one held the quality floor.

### Keep a prompt's turns together in the statistics

The trajectory comes from the promptId of the user turn that started the exchange, so every assistant message answering one prompt is clustered as a unit. The intervals are resampled at that level rather than treating a long agentic turn as a pile of independent trials.

### Audit an agent built on Claude Code

If your team ships automation on headless Claude Code (CI jobs, hooks, subagent fan-outs) the session logs are already complete traces. The audit reads them as they are, so a coding agent gets right-sized without standing up an observability stack first.

## The honest part

- It reads the JSONL session logs already on disk; it does not watch live sessions, and it never changes which model Claude Code uses. Acting on the report stays your decision.
- An assistant message has to carry a message id, a model, and a usage object. Rows missing any of those are dropped with a recorded reason rather than guessed at, and a message that changed model between rows is dropped too.
- Claude Code logs token counts, including cache reads and writes, but no dollar cost. Historical spend is not derived from them; candidate calls are priced from what your replay endpoint reports.
- Evidence minimums are per evaluator kind: at least 10 assessed executions, at least two distinct step IDs, and at least five distinct trajectories. Coding sessions are dense, so a short session usually abstains for want of distinct trajectories rather than producing a thin recommendation.
- Steps whose output can feed later model-authored steps are confirmed by running your pipeline in a container with the candidate swapped in. That needs a --modeb-config file naming the image and the step map; without it the recommendation is reported as unconfirmed.
- It is a report, not a gateway. Claude Code keeps calling its provider directly; rightmodeler never sits in the request path.

## FAQ

### Where does Claude Code keep its session logs?

In ~/.claude/projects, with one folder per project (the project path is encoded into the folder name) and one JSONL file per session. Pass a single file or a folder with --traces; the format is autodetected from the type, sessionId, and parentUuid keys.

### Do I need to export my traces first?

No, this is the one integration with no export step. rightmodeler is installed as a Claude Code skill, so it runs inside the tool that wrote the logs and reads them straight off disk.

### How does it turn a transcript into steps?

One assistant message is one step. Rows sharing a session and a message id are stitched back together, the user turns above it (up to the previous assistant) become its input messages, and tool_result blocks from those turns are attached to the output so the answer is scored with the evidence it was given.

### Does my code leave my machine?

Replays send step inputs, which for a coding session include your prompts and code, to candidate models through the endpoint you configured, under your own key, the same shape of call Claude Code already makes. There is no rightmodeler server and no account; nothing else holds your traces.

### Can it switch the model my sessions use?

Not on its own. It produces per-step verdicts with the evidence attached, and where a family clears every gate it opens a draft pull request against your repository that you review and merge. Nothing changes without that merge.

### Does it work on headless (claude -p) runs?

Yes. Headless runs write the same JSONL session format to the same directory, so the adapter reads them identically. CI-driven pipelines are the best-fit case, because they can also be run in a container for the confirmation pass.