Integration · Phoenix
rightmodeler + Phoenix
Phoenix already records your agent as OpenInference spans: model, messages, token counts. rightmodeler replays those spans through cheaper candidates and reports agreement, evidence, sample size, and abstentions per step family.
TL;DR
rightmodeler reads Phoenix traces in their native OpenInference form with a dedicated adapter. It walks an OTLP file export down through resourceSpans and scopeSpans, keeps the spans whose openinference.span.kind is LLM, un-flattens the indexed dotted message keys back into ordered messages, and reads the model from llm.model_name with token counts from llm.token_count. Each kept span is then replayed against cheaper candidates and scored against the output you already shipped.
How it works with Phoenix
Get spans out of Phoenix
Export your project's spans as an OTLP file, JSON or JSONL, keeping the resourceSpans envelope and each span's OpenInference attributes intact. That attribute map is all the adapter needs; save the file anywhere on disk.
Autodetected, then un-flattened
The adapter recognizes the format from an openinference.span.kind attribute alongside keys beginning llm., then keeps the spans whose kind is LLM. Messages arrive flattened into indexed dotted keys (llm.input_messages.0.message.role, one key per field), so it rebuilds them in index order, decodes JSON-string payloads, appends llm.tools.*.tool.json_schema as tool definitions, and reads the answer from llm.output_messages or output.value.
Replayed on your inputs, judged against your outputs
Each kept span is replayed against cheaper candidates through the OpenAI-compatible endpoint you name with --base-url and --api-key-env. A configured external evaluator scores the outputs when it is reachable; otherwise the built-in judge runs two temperature-zero calls with the reference and candidate positions swapped. Verdicts are confidence lower bounds kept separate per evaluator kind and never pooled across kinds.
A per-step report, then a pull request
The run writes report.md and report.json with a verdict per step family and streams JSONL events as it works. Thin evidence produces an abstention with a named reason. When a family clears every release gate, rightmodeler apply opens a draft pull request that changes model identifiers and nothing else, watch reconciles it against CI, and rollback restores the pre-apply files, refusing unless every restored digest matches.
Setup
Nothing to install: npx runs the published CLI. Hand it an exported span file. Nothing connects to your Phoenix instance; the audit runs offline on the spans you already collected, against whichever replay endpoint you configure.
# preview the pipeline against your spans, without spending
# then run it against your replay endpoint
# or let your coding agent drive it
What rightmodeler reads from Phoenix
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: openinference.span.kind · llm.model_name
Use cases
Right-size whatever OpenInference instruments
Phoenix traces frameworks through OpenInference instrumentors, so whether the spans came from LlamaIndex, LangChain, or your own tracer, every LLM span lands as a step with its model, messages, and token counts, and the audit shows which step families genuinely need the model they are on.
Measure the candidate before Phoenix sees drift
The usual order is swap first, then watch the dashboards. rightmodeler inverts it: replay recorded spans against the candidate offline, read the agreement with outputs your users already accepted, and review the evidence before changing a model.
Keep sessions intact in the statistics
A span's session.id becomes its trajectory, and trajectories are the clustering unit for the intervals. A five-span agent run over twenty sessions is resampled as twenty units rather than a hundred independent trials, which keeps the confidence bounds honest.
The honest part
- Only spans whose openinference.span.kind is LLM become steps. Chain, tool, and retriever spans are read for detection, then dropped with a recorded reason rather than replayed.
- It reads span files on disk, OTLP JSON or JSONL with OpenInference attributes. It does not connect to a running Phoenix instance, poll its API, or watch new traces arrive.
- The integration is with the OpenInference span format, not with Arize: no account connection and no keys exchanged. Spans that use gen_ai.* attributes instead are picked up by the separate OTel GenAI adapter.
- OpenInference spans carry no cost, so historical spend is not read from them. Candidate calls are priced from what your replay endpoint reports and marked as an estimate when the provider returns none.
- Evidence minimums are per evaluator kind: at least 10 assessed executions, at least two distinct step IDs, and at least five distinct trajectories. Below any of those the family abstains with a named reason instead of recommending.
- Steps whose output can feed later model-authored steps are confirmed by running your pipeline in a container with the candidate swapped in. That needs a --modeb-config file naming the image and the step map; without it the recommendation is reported as unconfirmed.
Frequently asked questions
How do I export traces from Phoenix?
Export your project's spans as OTLP JSON or JSONL, keeping the resourceSpans envelope and each span's OpenInference attributes intact. Pass the file or folder with --traces; the format is autodetected from the openinference.span.kind and llm.* attribute keys.
Does rightmodeler replace Phoenix?
No. Phoenix watches and evaluates your agent in production; rightmodeler runs after the fact, on exported spans, to measure cheaper candidates against accepted outputs. Different jobs, same traces.
Do I need an Arize account or API key?
No. The integration is with the OpenInference span format itself, so a self-hosted Phoenix works exactly like the cloud one: export the spans and hand over the file. The only credential the audit needs is the key for the replay endpoint you configure.
Why do my spans have keys like llm.input_messages.0.message.role?
That is OpenInference's convention: arrays are flattened into indexed dotted keys inside the span's attribute map. The adapter rebuilds the messages in index order and decodes JSON-string payloads, so you hand it the spans exactly as they were exported.
Does my span data leave my machine?
Only during replay: step inputs go to candidate models through the endpoint you configured, under your own key, the same class of call your agent already makes. There is no rightmodeler server and no account; the span files stay on disk.
Will it always find a cheaper model?
No, and deliberately so. A family abstains when the evidence is thin, when the excluded fraction runs over the ceiling, or when no candidate clears the quality floor again on held-out cases. A tool that always finds a swap is not measuring anything.
Run the audit on your own traces
The CLI runs from npx, nothing to install; your Phoenix traces do the rest.