How an AI gateway can invalidate your model evaluation.

You want to know whether a cheaper model can take over a step in your application. So you send the step’s recorded inputs to the candidate through the gateway that already fronts your traffic, grade what comes back, and read the score. Every call returns HTTP 200. Some of those answers were not written by the candidate.

A gateway exists to keep production answering: it retries on another backend, maps one name to several deployments, serves repeats from a cache, and adjusts requests a provider cannot take. Each of those is a feature in production and a contamination in an evaluation, which makes one claim per row: this model, given this request, produced this output. This guide walks through the four ways a gateway breaks that claim: a fallback, an alias, a cached answer and a rewritten request. Each is shown on a response from Portkey, Bifrost or Agent Router, formerly Envoy AI Gateway, captured from a running gateway or written from its pinned source, with the way to detect it and the fix.

Before a row can count toward a decision, three things must be true:

• The model you named answered. Not a backup, not an alias target, not a model the provider swapped in.
• It answered now. A stored answer says nothing about the candidate today, and its latency and cost are the cache’s.
• It answered the request you sent. If the gateway added text, removed a parameter or converted the call to another API, the candidate was graded on a request it never saw.

None of this shows up in the status code: a gateway’s job is to turn trouble into a 200. The evidence is in the response body and headers, reported differently by each gateway, and sometimes not at all. An evaluation route needs configuration that turns these features off, and a check on every response that catches what the configuration missed.

Every response below is a fixture in the open-source rightmodeler repository, labeled by how it was produced:

• Captured from a running gateway: Portkey 1.15.2 on 2026-09-23, against a local echo upstream that answers with the model it receives; Agent Router v1.1.0, run standalone with aigw run, on 2026-09-23; Bifrost v2.2.1 on 2026-09-24. Agent Router and Bifrost called Vercel AI Gateway, and Bifrost also OpenRouter.
• Written from pinned vendor source: markers the stock images cannot produce locally, in source-derived.json, each entry citing the vendor file and lines at the pinned tag.
• Hermetic stub test: provenance.test.ts runs the response check over every fixture, offline.
• Live provider test: opt-in suites run each pinned image in Docker against Vercel AI Gateway, with models discovered from its live catalog, through a full replay and judge pass under a spend cap. The integration pages record those runs on 2026-09-23.

The images are pinned by digest. Agent Router’s image keeps its Envoy name:

portkeyai/gateway:1.15.2
  sha256:97f094d9c8a764cbfaa2a7138c0017b247ca923bb06db1b4c13b7f8a33b5200d
envoyproxy/ai-gateway-cli:v1.1.0
  sha256:df69760bb46b6dcb8e9c6cc3cbf040d02e1b970dab05568c478fdcc418d144b6
maximhq/bifrost:v2.2.1
  sha256:a8942692af7b4b89196cd8fc33653b7353488dfd58b24078fe793b8574a8084b

What happens. When a route’s primary backend fails, the gateway sends the request to the next backend, often a different model, and returns that answer with a 200. Portkey’s fallback strategy triggers on any non-2xx status by default, and Agent Router’s documentation describes falling back from an expensive model to a less expensive one on the same provider. In an evaluation that does two kinds of damage: the fallback’s answer is graded as the candidate’s, and the candidate’s failure, which you needed to count, disappears.

The example. The captured route, written by capture-config.mjs with the acceptance kit’s aigw-config.mjs, answers the model name fallback-demo with two backends: a priority-0 backend, mock-500.mjs, that answers every call with HTTP 500, and a priority-1 backend on Vercel AI Gateway whose modelNameOverride sends amazon/nova-micro upstream. A BackendTrafficPolicy retries on HTTP 500 with one attempt per priority, which, as Agent Router’s fallback guide describes, is what moves a failed call to the next priority. The captured response, trimmed:

{
  "requestedModel": "fallback-demo",
  "status": 200,
  "headers": { "content-type": "application/json" },
  "body": {
    "model": "amazon/nova-micro",
    "choices": [
      {
        "message": {
          "role": "assistant",
          "content": "The town council approved the installation of …"
        }
      }
    ]
  }
}

The status is 200 and the body names amazon/nova-micro. Nothing else in the captured response marks the fallback; the model field is the only witness.

Fallbacks can also happen where the gateway cannot see them. Bifrost v2.2.1 reports a swap made inside a single provider call, which its source describes as Anthropic’s server-side fallback, in the server_side_fallback_model field of routing_info. In the entry written from that source, the response’s own model field still names the model you asked for:

"model": "anthropic/claude-sonnet-4.5",
"extra_fields": {
  "routing_info": {
    "provider": "anthropic",
    "model": "claude-sonnet-4.5",
    "server_side_fallback_model": "anthropic/claude-haiku-4.5"
  }
}

Evidence. Captured from a running gateway: envoy/fallback.json and envoy/plain.json. Written from pinned vendor source: the bifrost-server-side-fallback entry. Hermetic stub test: provenance.test.ts, in the Envoy fallback test and the Bifrost marker test. Live provider test: gateway-envoy.live.test.ts puts a server that answers every call with HTTP 500 at priority 0 and checks that every answer the candidate’s route returned is left out of the evidence.

Detect it. Compare the model each response names with the model you requested, and read the gateway’s own fallback markers. Names legitimately differ: Bifrost answers vercel/amazon/nova-micro as amazon/nova-micro, as bifrost/chat.json shows, and providers add dated snapshots such as gpt-4o-mini-2024-07-18 for gpt-4o-mini. A strict comparison flags those and a loose one lets substitutions through; the rule in provenance.ts accepts those two differences, ignoring letter case, and refuses another model name, another vendor segment, or a suffix that is not a date. The check also needs a model field, and Agent Router’s model name virtualization page notes that some upstreams, AWS Bedrock’s Converse API among them, return none. There, the route configuration has to carry the guarantee.

Fix it. Give the evaluation its own route: one backend per model under the upstream’s own id, with no priority fallback, no modelNameOverride and no retry policy that moves to another backend. On Portkey, send no config with fallback targets; on Bifrost, send no fallbacks array in the request body. Treat a failed call as the candidate’s result, not a gap to fill. Leave any answer from another model out of the evidence, and count it.

What happens. Every gateway here lets the name you send resolve to a different model: Portkey’s override_params, Agent Router’s modelNameOverride, Bifrost’s key aliases and routing rules. Bifrost’s documentation lists giving different teams different underlying models behind the same name as a use for aliases, and its routing-rule aliases apply per virtual key, team or customer. The same name can mean one model for your production key and another for your evaluation key.

The example. The Portkey 1.15.2 capture, capture.sh, routes to an echo upstream that answers with whatever model it receives, through an x-portkey-config whose override_params sets the model to stub/override. The response, trimmed:

{
  "requestedModel": "stub/requested",
  "status": 200,
  "headers": {
    "x-portkey-cache-status": "DISABLED",
    "x-portkey-last-used-option-index": "config"
  },
  "body": {
    "model": "stub/override",
    "choices": [
      { "message": { "role": "assistant", "content": "echo 48" } }
    ]
  }
}

The request asked for stub/requested and the upstream received stub/override. The same call with no config comes back naming stub/requested.

Evidence. Captured from a running gateway: portkey/override-params.json and portkey/plain.json. Hermetic stub test: provenance.test.ts, in the override_params test, beside the test that accepts Bifrost’s prefix-stripped answer. Live provider test: gateway-portkey.live.test.ts sends an x-portkey-config whose override_params names the dearer of two incumbent models, gateway-bifrost.live.test.ts maps a candidate to an incumbent with a key alias, and both check that every candidate answer is left out of the evidence.

Detect it. The served-model comparison from the first pitfall catches an alias whenever the upstream reports the model it ran. Portkey’s x-portkey-last-used-option-index reports which target of a config served the call; with a single target it reads config, as in both captures, so for override_params the model field is the witness. Do not infer from your request that no alias applied: Portkey’s docs note that a default config attached to an API key applies its routing, fallbacks and caching even when a request carries no x-portkey-config header.

Fix it. Name evaluation models by their upstream ids, configure no aliases, routing rules, overrides or configs for them, and run the evaluation with a key whose settings you have read.

What happens. A response cache returns a stored answer without calling the model: an exact cache for an identical request, a semantic cache for a merely similar one. The answer is not a fresh sample from the candidate, and its latency and cost belong to the cache, which flatters exactly the numbers a cost evaluation reads. A served-model check does not help, because a cached answer names the model that wrote it.

The details matter. Portkey’s semantic cache requires the model and every other body parameter to match exactly but ignores the system prompt, so an evaluation of a system-prompt change can be answered from an entry written under the old prompt. Bifrost keys its cache by model by default (cache_by_model: true); turned off, different models can share entries. And Bifrost’s x-bf-cache-no-store header skips writing the response but, in its docs’ words, “still serves cached hits”.

The example. Caching is compiled out of the stock Portkey image, whose conf.json sets cache to false, so the captured Portkey responses report x-portkey-cache-status: DISABLED. The hit markers are written from pinned vendor source: Portkey’s status values from src/middlewares/cache/index.ts at v1.15.2, and Bifrost’s cache_debug from core/schemas/bifrost.go at transports/v2.2.1:

x-portkey-cache-status: SEMANTIC HIT

"extra_fields": {
  "cache_debug": { "cache_hit": true, "hit_type": "semantic" }
}

Evidence. Written from pinned vendor source: the portkey-cache-hit, portkey-semantic-cache-hit and bifrost-cache-hit entries. Captured from a running gateway: portkey/plain.json, whose DISABLED status counts as fresh. Hermetic stub test: provenance.test.ts, in the Portkey cache-status test and the Bifrost marker test.

Detect it. Read the cache marker on every response. Portkey’s x-portkey-cache-status reports HIT or SEMANTIC HIT for a cached answer, and MISS, SEMANTIC MISS, REFRESH or DISABLED for a fresh one. Bifrost reports the hit in the response body, as extra_fields.cache_debug.cache_hit, and on a stream only the final chunk carries the full payload, so the check has to read the body, and the last chunk of a stream, not only the headers.

Fix it. Turn caching off on the evaluation route. Bifrost caches only when a request carries x-bf-cache-key or the plugin has a default_cache_key, so send no cache key, leave the default empty, and add x-bf-cache-no-store: true so evaluation answers never land in production’s cache. On Portkey, send no cache config. Leave out any hit that still arrives.

What happens. Some gateway features change the request on its way upstream: guardrail mutators that edit messages, and compatibility layers that drop parameters a model does not support or convert the call to another API. The model answers honestly, to a different question. This is the subtlest of the four: the served model is right and nothing was cached, yet the output is not evidence about the request you meant to test.

The example. Bifrost’s compat plugin drops parameters its model catalog does not list for a model and reports them in dropped_compat_plugin_params. The v2.2.1 capture sends response_format to openrouter/amazon/nova-micro-v1, whose catalog entry lists no structured output, under compat-drop-config.json, whose client block leaves every compat flag out. The response, trimmed:

{
  "requestedModel": "openrouter/amazon/nova-micro-v1",
  "status": 200,
  "body": {
    "model": "amazon/nova-micro-v1",
    "choices": [
      {
        "message": {
          "role": "assistant",
          "content": "The city mentioned in the note is Lisbon."
        }
      }
    ],
    "extra_fields": {
      "dropped_compat_plugin_params": ["response_format"]
    }
  }
}

A structured-output request came back as a prose sentence, with a 200. An evaluation that scores JSON validity would fail the candidate for ignoring a parameter it never received; one that scores content would pass it on a request with no schema.

The defaults are the trap. Bifrost’s config reference lists each compat flag as false by default as of 2026-09-24, while in the source at v2.2.1 a client block that omits a flag turns it on, and the capture matches the source. A request can also switch the plugin on with an x-bf-compat header.

Portkey reports its rewrites in hook_results. The third request in capture.sh sends a config with the default.addPrefix mutator, which prepends text to the user message. The response, trimmed:

"hook_results": {
  "before_request_hooks": [
    {
      "id": "input_guardrail_pod",
      "type": "mutator",
      "verdict": true,
      "transformed": true,
      "checks": [
        {
          "id": "default.addPrefix",
          "transformed": true,
          "data": {
            "prefix": "PREFIX-INJECTED: ",
            "applyToRole": "user"
          }
        }
      ]
    }
  ]
}

Portkey’s guardrail docs define transformed as whether a guardrail modified the request or response; in a stream, hook_results are hidden unless x-portkey-strict-open-ai-compliance is false.

Evidence. Captured from a running gateway: bifrost/compat-drop.json and portkey/input-mutator.json. Written from pinned vendor source: the bifrost-dropped-params, bifrost-dropped-tools and bifrost-converted-request entries. Hermetic stub test: provenance.test.ts, in the compat-drop test, the Portkey hook test and the Bifrost marker test.

Detect it. Treat any reported change as disqualifying: transformed: true in Portkey’s hook results, and dropped_compat_plugin_params, dropped_unsupported_tools or converted_request_type in Bifrost’s extra_fields.

Fix it. Set every compat flag to false explicitly, send no x-bf-compat header, and attach no guardrails or mutators to the evaluation route. The client block of the acceptance kit’s replay-safe bifrost/config.json:

"client": {
  "enable_logging": true,
  "compat": {
    "convert_text_to_chat": false,
    "convert_chat_to_responses": false,
    "should_drop_params": false,
    "should_convert_params": false,
    "azure_deepseek": false
  }
}

If a candidate cannot take response_format or tools, that is a finding about the candidate, not something for the route to smooth over.

An evaluation that grades candidates against the outputs your team already accepted, the approach called reference evidence, has a second exposure: those outputs came through the same gateway. If production traffic fell back, some references were written by the fallback model and recorded under the name your application asked for.

Agent Router shows the difference. Its tracing records the full request and response on each span by default, which makes the spans usable as references. On the evaluation route a priority-fallback answer is caught, because the response names another model. In exported traces it is not: the span recorded for the same captured fallback call, line 9 of envoy-openinference.jsonl, carries a request body naming fallback-demo, an llm.model_name of amazon/nova-micro, and no attribute that says a fallback chose it. A modelNameOverride alias produces the same pair of names, so the span cannot tell the two apart. A reader that takes the requested model from the request body, as rightmodeler’s does, records that output as fallback-demo’s, as its hermetic test asserts. The span does carry both names, which is enough to flag it: a span whose llm.model_name names another model than its request body was not answered by the model your application named, whether a fallback or an alias chose it. Line 9 is the only such span in the file. Keep fallback routes off the traffic you export as references.

Bifrost’s log store keeps the difference visible: a fallback attempt is its own row with a fallback_index above 0, so rightmodeler’s Bifrost reader leaves it out as fallback_answer and the failed primary as call_failed. The hermetic bifrost.test.ts checks this on a log export captured from v2.2.1 on 2026-09-24, and the Bifrost live provider test on a fresh one. Export without roots_only=true, which Bifrost’s API reference describes as collapsing fallback rows into their root.

• One backend per evaluated model, named by the upstream’s own id.
• No fallbacks, priority backends, or retries that move to another backend.
• No aliases, overrides, routing rules or configs on the evaluation’s model names or key.
• Caching off, and a no-store header on every evaluation call where the gateway has one.
• No guardrail mutators; compatibility flags set to false explicitly.
• For every response, its named model, cache marker and rewrite markers recorded, and every response that fails a check left out and counted.
• No verdict for a model with more than a small share of its responses left out, until the route is fixed and the evaluation rerun.
• References exported only from routes without fallbacks, or from a log store that marks fallback rows.
• Evaluation traffic tagged, and left out of the next export.

rightmodeler is an MIT-licensed CLI on npm (npx rightmodeler init) that reads the traces you already export, replays recorded steps through cheaper candidates from your provider’s live catalog, and judges them against the outputs your team already accepted, reporting reference agreement, sample size and abstentions against a held-out quality floor. It is never in the request path: its output is a draft pull request that changes only model identifiers, for a human to review and merge. It replays through any OpenAI-compatible base URL, including each gateway above.

Every replayed and judge response goes through the check in provenance.ts before it counts. A response that names another model, reports a cache hit or reports a changed request is recorded as substituted, never graded, and counted as attribution_substituted. If more than 5% of a family’s replays are left out of the evidence, the family abstains instead of deciding. A replay_responses_substituted warning names the requested and served models and the fix, and a judge that answers as another model is retired while the next-ranked judge takes over. The replay-safe setup for each gateway is on the Portkey, Bifrost and Agent Router integration pages, and the method is on how it works.

The hermetic checks need no network and no keys. From a clone of the repository:

pnpm install --frozen-lockfile
pnpm --filter @rightmodeler/replay exec vitest run src/provenance.test.ts

The live provider tests need Docker and an AI_GATEWAY_API_KEY for Vercel AI Gateway, and each leg runs under a spend cap of $0.25 or less. The fixtures README maps each pitfall to its fixture, tests and evidence label, with the exact commands: Reproducing the gateway evaluation pitfalls.