Integration · Phoenix
rightmodeler + Phoenix
Phoenix already records your agent as OpenInference spans: model, messages, token counts. rightmodeler replays those spans through cheaper candidates and reports, per step, which swaps hold quality.
TL;DR
rightmodeler reads Phoenix traces in their native OpenInference span format with a dedicated adapter: it detects openinference.span.kind and llm.* attributes, un-flattens the dotted message keys, and folds every span into one per-step schema. Each successful step is then replayed through cheaper candidates on OpenRouter and judged against the output you already shipped, right-sizing without new instrumentation.
How it works with Phoenix
Get spans out of Phoenix
Query spans from your Phoenix project (get_spans_dataframe() in the Phoenix client, or any span query) and save the result as JSON or JSONL on disk, one file or a folder. Each record should keep its OpenInference attributes; that attribute map is all the adapter needs.
Autodetected, then un-flattened
The ingester recognizes the format by openinference.span.kind and llm.* attribute keys and applies the dedicated OpenInference adapter. Messages arrive flattened into indexed dotted keys (llm.input_messages.0.message.role, one key per field) so it un-flattens them back into ordered messages and folds every span into one per-step schema.
Replayed on your inputs, judged against your outputs
Each successful LLM step is replayed through cheaper OpenRouter candidates on the exact messages recovered from your spans. A judge from a different model family scores every candidate against the output you already shipped, position-swapped, with a 0.90 quality floor by default. Steps with tool calls or loops are confirmed by re-running your pipeline code end-to-end instead.
A per-step report, never all-or-nothing
The report lists each proposed swap with its evidence and confidence, and the TUI walks you through approve, reject, or hold, one step at a time. High-risk families (auth, payments, migrations, deploys) are flagged and stay on the current model, and weak evidence means abstain.
Setup
Install once, then hand the skill an exported span file. Nothing connects to your Phoenix instance; the audit runs offline on the spans you already collected, with your own OpenRouter key for replays.
# install the rightmodeler skill
# inside Claude Code, hand the exported spans to the skill
What rightmodeler reads from Phoenix
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: openinference.span.kind · llm.model_name
Use cases
Right-size whatever OpenInference instruments
Phoenix traces frameworks through OpenInference instrumentors, so whether the spans came from LlamaIndex, LangChain, or your own tracer, every LLM call lands as a step with its model, messages, and token counts, and the audit shows which steps genuinely need the model they are on.
Prove the downgrade before Phoenix sees drift
The usual order is swap first, then watch the dashboards. rightmodeler inverts it: replay recorded spans through the candidate model offline, read the judge's verdict against outputs your users already accepted, and ship only what proved out.
Put a price on spans that carry none
OpenInference spans record token counts but no dollar figures. The audit derives each step's cost baseline from llm.model_name and llm.token_count.* via OpenRouter's pricing table, then itemizes what each step family costs today and which proven swaps would change that.
The honest part
- It reads span files on disk, OTLP JSON or JSONL with OpenInference attributes. It does not connect to a running Phoenix instance, poll its API, or watch new traces arrive.
- The integration is with the OpenInference span format, not with Arize: no account connection and no keys exchanged. Spans that use gen_ai.* attributes instead are picked up by the separate OTel GenAI adapter.
- OpenInference spans carry no cost, so the baseline is derived from the model name and token counts against OpenRouter's pricing table, a consistent estimate, not your provider invoice.
- Steps with tool calls or loops are not judged from the span alone; they are confirmed by re-running your pipeline end-to-end in a throwaway worktree with the candidate model injected.
- It is a report, not a runtime gateway. Phoenix keeps observing your agent in production; rightmodeler never sits in the request path, and where evidence is weak it abstains.
Frequently asked questions
How do I export traces from Phoenix?
Query spans with the Phoenix client (get_spans_dataframe() or a span query scoped to your project) and save the result as JSON or JSONL with each span's OpenInference attributes intact. Point the skill at the file or folder; the format is autodetected.
Does rightmodeler replace Phoenix?
No. Phoenix watches and evaluates your agent in production; rightmodeler runs after the fact, on exported spans, to prove which steps can move to a cheaper model. Different jobs, same traces.
Do I need an Arize account or API key?
No. The integration is with the OpenInference span format itself, so a self-hosted Phoenix works exactly like the cloud one: export the spans and hand over the file. The only key involved is your own OpenRouter key, used for replays.
Why do my spans have keys like llm.input_messages.0.message.role?
That is OpenInference's convention, arrays are flattened into indexed dotted keys inside the span's attribute map. The ingester un-flattens them, rebuilding ordered messages by index, and parses JSON-string payloads defensively, so you hand it the spans as they are.
Does my span data leave my machine?
Only during replay: step inputs go to candidate models through OpenRouter under your own key, the same class of call your agent already makes to a provider. There is no rightmodeler server and no account; the span files themselves stay on disk.
Will it always find a cheaper model?
No, and deliberately so. Where evidence is weak the audit abstains, and high-risk step families (auth, payments, migrations, deploys, secrets) are flagged and left on the current model. A tool that always finds a swap is not measuring anything.
Run the audit on your own traces
The skill is free on GitHub. One command installs it; your Phoenix traces do the rest.