Source: https://www.rightmodeler.com/integrations/braintrust

# rightmodeler + Braintrust

Braintrust scores your agent against your evals. rightmodeler puts the same span rows to a different test: which steps survive on a cheaper model, judged against the outputs you already shipped, and scored by Braintrust itself when you point it there.

rightmodeler has a dedicated Braintrust adapter. It recognizes span rows by their span_id, root_span_id, and span_attributes keys, maps the rows whose span_attributes.type is llm, and takes the model from metadata.model with token counts from metrics. Braintrust can also be the evaluator: pass --evaluator braintrust with a project id and scorer names and your own scorers grade the replays, with the built-in judge used only when nothing better is reachable.

## How it works with Braintrust

### 01 · Fetch: Save span rows to disk

Pull rows from the REST API with your BRAINTRUST_API_KEY: GET /v1/project_logs/{project_id}/fetch for live logs, or /v1/experiment/{experiment_id}/fetch for experiment runs. The response nests rows under an events array; save that array (jq .events) as a JSON file anywhere on disk.

### 02 · Ingest: Autodetected, then mapped field by field

The adapter recognizes the rows by span_id, root_span_id, and a span_attributes object, then keeps the ones whose span_attributes.type is llm. Each kept row contributes its model from metadata.model, its messages from input, its answer from output, and its token counts from metrics.prompt_tokens and metrics.completion_tokens. Rows sharing a root_span_id become one trajectory, ordered by metrics.start.

### 03 · Replay + score: Cheaper candidates, graded by your own scorers

Each kept step is replayed against cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env. With --evaluator braintrust, your project's scorers grade the replays and their pass decisions feed the release gate; without a reachable evaluator the run warns and falls back to the built-in judge, which makes two temperature-zero calls with the reference and candidate positions swapped. Verdicts stay separate per evaluator kind and are never pooled.

### 04 · Report + apply: A report, then a pull request you review

The run writes report.md and report.json with a verdict per step family and streams JSONL events as it works. Where the evidence is thin the family abstains with a named reason. When one clears every release gate, rightmodeler apply opens a draft pull request that touches model identifiers and nothing else, and rollback restores the pre-apply files, refusing unless every restored digest matches. rightmodeler export --to braintrust pushes the trials and verdicts back into the project you already review.

## Setup

Nothing to install: npx runs the published CLI. Point it at your saved span rows. The audit reads the file offline; your Braintrust key is needed only for the fetch you run yourself, and again if you choose Braintrust as the evaluator.

```bash
npx rightmodeler init
```

```bash
# # fetch span rows, save the response's events array as your trace file
curl -s https://api.braintrust.dev/v1/project_logs/$ID/fetch -H "Authorization: Bearer $BRAINTRUST_API_KEY" | jq .events > braintrust.json
```

```bash
# # score the replays with your own Braintrust scorers
npx rightmodeler init --traces ./braintrust.json --base-url $BASE_URL --api-key-env PROVIDER_API_KEY --evaluator braintrust --evaluator-project-id $ID --evaluator-scorer factuality
```

```bash
# # push the trials and verdicts back into Braintrust
npx rightmodeler export --to braintrust --project-id $ID
```

## What rightmodeler reads from Braintrust

| Normalized field | Source field |
| --- | --- |
| trajectory | root_span_id |
| step family | span_attributes.name |
| model | metadata.model |
| input messages | input |
| output | output |
| token usage | metrics.prompt/completion_tokens |
| ordering | metrics.start, else created |

## Autodetection

Detected by: `span_id · root_span_id · span_attributes`

## Use cases

### Price the agent your evals already passed

Braintrust tells you the agent clears the bar you set; rightmodeler tells you which of its steps are overpaying to clear it. Same span rows, different question, answered with per-step evidence instead of a hunch.

### Grade the audit with the scorers you already trust

Point the run at Braintrust with --evaluator braintrust and your own scorers decide whether a candidate passed, rather than a judge model you did not write. The pass decisions, metric names, and rubric versions land in the ledger next to each trial.

### Build the corpus from a curated dataset

rightmodeler corpus import --from braintrust:dataset pulls a curated dataset instead of raw traces. Curated references are usually human-verified, and reference correctness is the ceiling on every number downstream, so a verified corpus raises what the audit can prove.

### Turn a finished experiment into a swap decision

Fetch the span rows from an experiment you already ran and replay its llm spans against cheaper candidates. The next model decision starts from evidence you already paid to generate.

## The honest part

- Only rows whose span_attributes.type is llm become steps. Score and task spans are read, then dropped with a recorded reason, so filtering the export to llm spans mostly saves you time rather than changing the result.
- The model has to be in metadata.model and the token counts in metrics. A row that records them somewhere else is dropped with its reason rather than guessed at.
- It reads exported rows on disk. It does not connect to your Braintrust project, poll the API, or watch new spans arrive; the evaluator and export paths are the only ones that call Braintrust, and only when you ask for them.
- Evidence minimums are per evaluator kind: at least 10 assessed executions, at least two distinct step IDs, and at least five distinct trajectories. Below any of those the family abstains with a named reason instead of recommending.
- Steps whose output can feed later model-authored steps are confirmed by running your pipeline in a container with the candidate swapped in. That needs a --modeb-config file naming the image and the step map; without it the recommendation is reported as unconfirmed.
- It is a report, not a runtime gateway. Braintrust keeps scoring your agent in production; rightmodeler runs offline on the files you fetched and never sits in the request path.

## FAQ

### How do I get span rows out of Braintrust?

Fetch them from the REST API, GET /v1/project_logs/{project_id}/fetch for live logs, /v1/experiment/{experiment_id}/fetch for experiment runs, with an Authorization: Bearer header carrying your BRAINTRUST_API_KEY. The response nests rows under events; save that array as JSON and pass the file with --traces.

### Does rightmodeler replace my Braintrust evals?

No, and it can run on top of them. Evals measure whether your agent clears a bar you defined; rightmodeler measures whether a cheaper model matches what you already shipped. With --evaluator braintrust your scorers are what decides, so the two answers are produced by the same rubric.

### How do I make Braintrust the evaluator?

Pass --evaluator braintrust with --evaluator-project-id and at least one --evaluator-scorer. The key comes from BRAINTRUST_API_KEY unless --evaluator-api-key-env names another variable. Add --evaluator-gate-metric when several scorers are configured, and --evaluator-gate-threshold when the evaluator returns no pass decision of its own.

### Can I import a Braintrust dataset instead of traces?

Yes. rightmodeler corpus import --from braintrust:<dataset> builds the replay corpus from a curated dataset, recording each case as a curated reference. That matters because the correctness of the reference caps every downstream number, and curated datasets usually carry human-verified answers.

### Does my span data leave my machine?

Replays send step inputs to candidate models through the endpoint you configured, under your own key, the same shape of call your agent already makes. There is no rightmodeler server and no account; Braintrust is contacted only by your own fetch, and by the evaluator or export commands when you run them.

### Will it always find savings?

No, by design. A family abstains when the evidence is thin, when the excluded fraction runs over the ceiling, or when no candidate clears the quality floor again on held-out cases. An audit that always finds a swap is not measuring anything.