Integration · Bifrost
rightmodeler + Bifrost
Bifrost is an Apache 2.0 AI gateway written in Go. Its open-source build can keep every call in a log store that rightmodeler reads as traces, and the same gateway can be the route the audit replays and judges through.
TL;DR
rightmodeler reads Bifrost's request logs, exported from its management API, as one ordered run per session: the model your application asked for, the conversation as sent, the output, usage, cost, latency and retries. The same gateway is a replay route: model ids are <provider>/<upstream id>, the billed cost comes from usage.cost.total_cost, and --catalog-reference fills the pricing a custom provider does not list. Turn every compat flag off and configure no aliases for replay models; a response from another model, a semantic-cache hit, or a request the compat plugin changed is left out of the evidence instead of graded. Bifrost advertises sub-millisecond overhead and runs by other gateway makers measured milliseconds, so test it on your own traffic: rightmodeler records each replay's latency through your gateway. Verified on the open-source build, v2.2.1.
How it works with Bifrost
Run a replay-safe gateway
Run maximhq/bifrost:v2.2.1 with a config.json whose client block sets every compat flag to false and keeps request logging on, and that gives replay models no key aliases or routing rules. A client block that omits the compat flags turns them all on, and the compat plugin can drop parameters such as response_format while still answering 200. Vercel AI Gateway works as an OpenAI-typed custom provider, so vercel/openai/gpt-4o-mini reaches openai/gpt-4o-mini upstream.
Export the request logs
From your application, send x-bf-session-id so the calls of one conversation form one run, and x-bf-dim-rightmodeler-family: <name> to name each call's family. Export chat completions from GET /api/logs oldest first, write one GET /api/logs/{id} detail per line, and pass the file with --traces. The Bifrost reader is picked automatically.
Only production answers become cases
Each row gives the model the application asked for (provider/model, or provider/alias when it sent an alias), the conversation from input_history, the output from output_message, and its token usage, cost, latency and retries. Failed calls, answers from a configured fallback, rightmodeler's own tagged replays, calls still running at export, and calls whose content was not logged are left out with a trace_steps_excluded warning that names each reason.
Replayed through Bifrost, graded only when fresh
Pass --catalog-reference with the upstream's public model list, and send x-bf-cache-no-store: true and x-bf-dim-rightmodeler: replay with --header. Candidates replay through POST /v1/chat/completions, the billed cost is read from usage.cost.total_cost, a gateway id such as vercel/openai/gpt-4o-mini takes its family from the vendor segment, and catalog entries that do not output text are dropped. A response that names another model, reports a server-side fallback or a cache hit, or reports that the compat plugin dropped parameters or converted the request is recorded as substituted and never graded.
Setup
Put a replay-safe config.json (every compat flag false, logging on, no aliases for replay models; rightmodeler docs gateways lists the rules) in a named volume, export the logs, then replay through the same gateway. The example uses Vercel AI Gateway as a custom provider.
# the upstream key, which config.json reads as env.AI_GATEWAY_API_KEY
# a named volume for config.json and the log store
# the open-source image, reachable from this machine only
# copy in the replay-safe config and start the gateway
# export chat completions oldest first; past 1000 rows, repeat with offset=1000 and append
# Bifrost holds the upstream key, so this only has to be non-empty
# replay through Bifrost: replays skip its cache and are tagged for the next export
# or let your coding agent drive it
What rightmodeler reads from Bifrost
Every run is folded into one per-step schema, the same one the replay and the judge run on.
autodetected by: input_history · provider · model · fallback_index · number_of_retries
Use cases
Traces without instrumenting every app
With logging on, every call through Bifrost lands in its log store as the application sent it, with cost and latency beside it. Export that store and the audit has its corpus, whatever language each service is written in.
Keep fallback answers out of the corpus
Bifrost logs a fallback attempt as its own row, so rightmodeler can tell a fallback's answer from the requested model's. Those rows, and the failed primary calls before them, are left out by name instead of becoming references a candidate is measured against.
Time the gateway on your own traffic
Bifrost's own overhead benchmark and the competing runs that disagree with it were all measured against mock upstreams. Every replay rightmodeler sends through your Bifrost records its latency, gateway hop included, next to the model and the billed cost, so you see how long real calls take through the gateway on your prompts and upstreams. rightmodeler does not separate Bifrost's own share.
The honest part
- Bifrost advertises sub-millisecond overhead: its own benchmark, on mocked OpenAI calls, reports 11 microseconds on a t3.xlarge at 5,000 requests per second, excluding JSON marshalling and the HTTP call. Runs by two other gateway makers measured milliseconds. BerriAI, which makes LiteLLM, reported 4.5 ms added at p99 on Bifrost v1.6.4 against a local mock (2026-07-22), and ENTERPILOT, which makes GoModel, reported 4.30 ms at p50 and 20.50 ms at p99 on v2.2.1 against a zero-latency mock (2026-09-18). rightmodeler records each replay's latency through your gateway, hop included, but does not separate Bifrost's share.
- Through Vercel AI Gateway as a custom provider, Bifrost gives every model the same placeholder created date. rightmodeler takes the release date from --catalog-reference instead, so judges rank as they do on Vercel: usually a large, recent model whose calls cost far more than a cheap candidate's replays. Set --max-cost-usd with that in mind.
- Set every compat flag to false and give replay models no key aliases or routing rules, or their replays are left out as substituted. A custom provider lists ids and context windows only, so pass --catalog-reference; Bifrost's own pricing sheet is not used.
- The OpenTelemetry plugin's export reads through the OTel GenAI reader, but it summarizes messages and loses tool-call ids, so use the log export for anything beyond plain text calls. Steps whose conversation contains tool calls are read but not replayed yet.
- Streamed calls through Mode B are checked on the stream's model and headers only. Bifrost reports a semantic-cache hit only in the response body, so such a hit on a streamed Mode B call is not detected. Mode B on the cloud backend cannot reach a gateway on localhost; use the Docker backend.
- Verified live on 2026-09-23 on maximhq/bifrost:v2.2.1, pinned by digest, with Vercel AI Gateway as a custom provider: the log export read as one ordered run per session with a failed call and a fallback answer named, every replayed response came from the requested model at the billed cost, and a key alias that mapped the candidate to another model had every replay left out as substituted. Enterprise features ship in a separate image and were not tested; pin the open-source image, because releases arrive weekly.
Frequently asked questions
Does Bifrost really add only microseconds?
That is Bifrost's own figure, measured on mocked calls excluding JSON marshalling and the HTTP call. Runs by two other gateway makers, BerriAI and ENTERPILOT, measured milliseconds against mock upstreams, and each of them makes a competing gateway. rightmodeler records every replay's latency through your Bifrost, so you can judge on your own traffic.
Why must the compat flags be off?
The compat plugin can drop request parameters, such as response_format, or convert the request to another type, and still answer 200, so a candidate would be graded on a request it never saw. rightmodeler leaves any response that reports dropped parameters or a converted request out of the evidence, and a config with every compat flag false keeps that from happening.
How do I export Bifrost's logs?
List chat completions with GET /api/logs?objects=chat_completion,chat_completion_stream&order=asc&limit=1000, then write each GET /api/logs/{id} detail as one line. Past 1000 rows, repeat with offset=1000, 2000 and so on, appending, until a page returns fewer rows; order=asc keeps earlier pages in place while new calls are logged. Do not pass roots_only=true, which hides fallback rows. Add credentials to the calls once your Bifrost has an admin.
Why does rightmodeler need --catalog-reference with Bifrost?
An OpenAI-typed custom provider, which is how Bifrost reaches Vercel AI Gateway, lists ids and context windows without prices or capabilities. rightmodeler joins each gateway id to the upstream's public list by its suffix, so vercel/openai/gpt-4o-mini takes openai/gpt-4o-mini's price, reading the list without your key.
How are rightmodeler's own replays kept out of the next export?
Pass --header 'x-bf-dim-rightmodeler: replay'. Bifrost stores it in the log row's metadata, and the next ingest leaves those rows out as replay_traffic. x-bf-cache-no-store: true keeps replays out of a semantic cache; a hit that still happens is detected and left out.
Does it work with Bifrost Enterprise?
Only the open-source build has been tested, at v2.2.1. Enterprise features such as guardrails, adaptive load balancing and clustering ship in a separate image and are not covered here.
Run the audit on your own traces
The CLI runs from npx, nothing to install; your Bifrost traces do the rest.