LLM cost optimization tools: which approach fits your workload?

Search for LLM cost optimization tools and you get a list: gateways, routers, caches, compressors, fine-tuning platforms. The list hides the useful part. Each category fixes a different problem, and a tool pointed at a problem you do not have costs engineering time and saves nothing, or saves money by quietly lowering quality.

This guide sorts six approaches by the problem each one solves. For each: what you need before you start, the evidence to have before it reaches production, and what it trades away. Then how they stack, because the order matters.

A disclosure first: we build rightmodeler, a tool for the first of these approaches, and we say where it fits. Vendor details come from each vendor’s own documentation, as of 2026-09-24.

Each approach attacks a different slice of the bill, so slice it first: by step, meaning each place your code calls a model, and by token type. The traces you already record usually hold everything you need. Five questions sort most workloads:

• Which steps run your most expensive model, and what does each of those steps actually do?
• Does one step receive a wide mix of easy and hard requests, or roughly the same kind every time?
• How much of each request repeats: a long system prompt, tool definitions, reference documents, the conversation so far? How many requests are exact duplicates?
• How much of the input does the step need? Tool output, logs and retrieved documents are often the bulk of it.
• Which jobs need an answer now, and which could wait an hour or a day?

In agent workloads the answers skew toward input, because the whole context is re-read on every turn. A study we wrote up in nobody can predict the bill measured agentic coding at about 154 input tokens for every output token.

A pipeline starts on one strong model, because that was the fastest way to make it work, and every step inherits it. Months later, the step that extracts a date and the step that plans a multi-tool task run the same frontier model at the same price.

Substitution fixes this statically: measure whether a cheaper model does each step’s job, and change the model identifier where it does. Nothing new enters the request path, and the saving lands on every call to that step.

Prerequisites: step-level traces with each call’s inputs and the outputs your team accepted, current prices for candidate models, and a way to judge whether a candidate agrees with the accepted output. Public benchmarks such as Artificial Analysis and Vals AI help you shortlist candidates; they cannot tell you how a model does on your step, with your prompts.

Evidence before rollout

• Agreement per step, not per pipeline. An average hides the one step that regressed.
• The case count beside every rate. A rate from a dozen cases is a hint, not evidence.
• A held-out check. Pick the best of many candidates on the same cases and part of its score is luck, so confirm the winner on cases it was not chosen on.
• A rule to abstain. When the evidence is thin, keep the current model.
• An independent judge from a different model family than either model being compared, or deterministic checks where the output has a fixed shape.

What it trades away: one model per step, chosen in advance, so it cannot adapt to a hard request inside an easy step. The accepted output is a reference, not ground truth, so agreement means the candidate matches what you shipped. And models and prices keep changing, so the measurement has to be repeatable.

This is the approach rightmodeler automates. It is an MIT-licensed CLI (npx rightmodeler init) that reads the traces you already export from tools such as Langfuse, Helicone or OpenTelemetry, resends recorded conversations to cheaper candidates from your provider’s live model catalog, and has a judge from outside both model families grade each output against the one you accepted. The report gives each step family’s pass count out of its trials, a worst-case bound, and an abstention reason wherever the evidence is too thin to recommend. A candidate chosen on one part of your traces must also clear your quality floor on a held-out part. Approved swaps arrive as a draft pull request that changes model identifiers only; a person reviews and merges it, and nothing runs in your request path.

For what that evidence looks like on a real system, the B:Side Assist case study follows 11 AI workloads from a single model and reasoning setting to a per-workload policy: a projected 70.8% lower inference cost, with the quality pass rate measured at 100% on a 20-query benchmark against the outputs B:Side had accepted. The reduce LLM costs page covers the workflow end to end.

Sometimes the variation is inside a step. A support assistant’s single entry point gets both password resets and multi-account billing disputes. Pin it to a cheap model and the hard cases fail; pin it to a strong one and the easy ones overpay.

A router makes the choice per request, at runtime. Not Diamond describes its router as analyzing each input and predicting which candidate model gives the best response at the lowest cost. Besides pre-trained routers, it offers custom routers trained on your own evaluation data: representative inputs, each candidate’s responses to them, and a score for every response.

That requirement is the one to plan for. A router is only as good as the evaluation scores it learns from, so routing needs the same per-candidate evidence as substitution, plus a runtime component. Before rollout, evaluate it on held-out traffic: quality per type of request, each model’s share of traffic, the cost, and what happens when the router or a candidate model is unavailable.

What it trades away: a live dependency in the request path, and harder debugging when one step is answered by different models on different days. Where a step’s traffic is roughly uniform, substitution gets most of the saving with neither. Our comparisons with Not Diamond and Martian go further. Martian today describes itself as an AI research lab whose Gateway offers one API to more than 200 models, with the model named on each request.

Caching is two different mechanisms that share a name, and they carry different risks.

Provider prompt caching: repeated prefixes

Most agent requests start with the same long prefix: system prompt, tool definitions, reference material, the conversation so far. Prompt caching lets the provider reuse its work on that prefix at a discount; the model still generates a fresh answer.

At OpenAI, prompt caching is on by default for supported models, and reused tokens are billed at a cached-input rate discounted by up to 90%. For GPT-5.6 and later, the minimum cacheable prefix is 1,024 tokens, a cache write costs 1.25 times the uncached input rate, and a read costs 0.1 times it. At Anthropic, a cache_control field turns caching on, once at the top level or on individual blocks, with a five-minute default lifetime and a one-hour option. Five-minute writes cost 1.25 times the base input price and reads 0.1 times it on most models. Both providers state that prompt caching does not change output generation.

Prerequisites: stable content first and changing content last. Both providers match on the exact prefix, so a timestamp near the top of a system prompt can defeat the whole cache. Anthropic also sets a minimum cacheable length per model, from 512 tokens on models such as Claude Opus 5.5 to 4,096 on Claude Haiku 4.5, and a shorter prompt is processed without caching.

Evidence before rollout is about cost only, because the output does not change: read the cache fields each provider returns with usage (cached_tokens at OpenAI, cache_read_input_tokens and cache_creation_input_tokens at Anthropic) and confirm the cached share of input rose. What it trades away is little beyond prompt restructuring, plus a write premium on prefixes that never get reused.

Gateway response caching: identical requests

A response cache sits in a gateway and returns a stored answer without calling the model at all. OpenRouter treats two requests as identical when the API key, model, endpoint type, streaming mode and request body all match; cache hits are not billed, and the default lifetime is five minutes, configurable up to 24 hours. Helicone hashes the request URL, body and relevant headers, with a default lifetime of seven days. LiteLLM offers exact-match caches keyed on the whole request, and semantic caches that serve the closest earlier match above a similarity threshold. Portkey offers both too; its semantic cache ignores the system prompt when matching and is available on select Enterprise plans.

The problem this solves is narrow: truly repeated requests, such as a public FAQ assistant, a classifier fed the same inputs, or a test suite run during development. Agent traffic rarely qualifies, because the context grows on every turn. LiteLLM’s own documentation warns that semantic caches suit single-shot prompts and go badly wrong on agentic traffic.

Evidence before rollout: the share of exact duplicates in your traces within the cache lifetime, which caps what an exact-match cache can save. For a semantic cache, sample its hits and check that each served answer fits the new request; a wrong near match is a quality failure no cost report shows. What it trades away: freshness, and for semantic caching, correctness at the margin. Our comparisons with Helicone, Portkey, LiteLLM and OpenRouter cover what else each gateway does.

In agent workloads, much of the input is tool output: logs, test results, search hits, documents read in full to find one line. The model pays to read it on the turn it arrives and on every turn after.

Compression cuts that input before the model sees it. On the provider side, Anthropic’s context editing clears old tool results once context passes a threshold you set, and OpenAI and Anthropic both offer compaction, which replaces earlier conversation with a shorter representation. Codag works on the tool output itself: by its own description, it reduces large results such as logs, test and build output and search results to the evidence the agent needs, and passes source code, diffs and configuration through unchanged. Mentlio works at the team level: by its own description, it measures locally what a team’s AI use costs and produces, and cuts token waste while keeping prompts and source code on the device.

Prerequisites: know what share of each step’s input is tool output or old history. Evidence before rollout: task success on the compressed inputs, not token counts, because a model cannot use what was removed. Then measure cost net of caching, since compaction and clearing change the prefix and both providers note this can reduce prompt-cache reuse. OpenAI’s guidance is to compare total input cost before and after, because fewer input tokens can still save money when the cache-hit rate falls; Anthropic’s tool-result clearing takes a clear_at_least setting so each clearing removes enough tokens to be worth the broken cache.

What it trades away: information, on every turn after the cut. Our comparisons with Codag and Mentlio go into where each one sits.

Evaluations, backfills and nightly classification do not need an answer in two seconds, and both major providers sell the same models at half price for work that can wait.

OpenAI’s Batch API charges 50% less than the synchronous APIs, runs against a separate and larger rate-limit pool, and completes each batch within 24 hours, often sooner. Anthropic’s Message Batches API charges 50% of standard prices, finishes most batches in under an hour, and expires requests not processed within 24 hours. Its prompt-caching multipliers stack with the batch discount, although cache hits inside a batch are best-effort.

Prerequisites: an asynchronous pipeline, a unique ID on every request, and handling for requests that expire. Evidence before rollout is operational, since the model and prompt are unchanged: completion times inside your deadline, and a working path for expired requests. What it trades away: latency, so it only fits jobs where nobody is waiting.

Some steps are narrow, high-volume and stable, such as routing a ticket to one of forty queues. The current solution is often a large model with a long prompt full of examples. Training a smaller model on the task, including on a larger model’s outputs, which is what distillation means, can shrink both the model and the prompt.

OpenAI’s model optimization guide lists those benefits: shorter prompts with fewer examples, and a smaller, cheaper, faster model trained for a task where a larger model is not cost-effective. As of 2026-09-24, OpenAI is also winding down its self-serve fine-tuning platform. Its deprecations page says organizations that had not fine-tuned before can no longer start, active existing customers lose the ability to create new jobs on January 6, 2027, and inference on fine-tuned models continues until the base model is deprecated. The other routes are training an open-weight model yourself or working with a vendor: Agnost AI, per its Y Combinator profile, turns an agent’s production conversations into custom models, and ThirdBrain Labs helps domain experts train models they own.

Prerequisites: a clean dataset of inputs and accepted outputs, an evaluation set held back from training, somewhere to serve the model, and a plan for retraining when the task drifts. Evidence before rollout: the trained model against the current one on held-out inputs from real traffic, and a lifecycle cost that includes training, hosting and retraining. What it trades away: the most upfront effort of any approach here, and a model you now maintain. Our comparisons with Agnost AI and ThirdBrain Labs go deeper.

The approaches combine, and the order changes what each is worth.

• Take the quality-neutral wins first. Prompt caching leaves the output unchanged, and batching runs the same model on the same prompt. Neither needs a quality evaluation, only proof the cost moved.
• Compress, then re-measure the model choice. A model choice measured on uncompressed traces describes a workload that no longer exists, so run substitution on traces recorded after the change. A step that needed a frontier model to find one failing test in a long log may not need it once the log is reduced to that test.
• Substitute per step, route where difficulty varies. Substitution covers steps with uniform traffic; a router earns its place only where traffic is mixed.
• Price candidates at your cache mix, not at list price. A model switch starts from a cold cache: OpenAI lists the model among the settings that affect the cached prefix. Cacheable minimums differ too. At Anthropic, a 2,000-token prefix that caches on Claude Sonnet 5, with its 1,024-token minimum, falls below Claude Haiku 4.5’s 4,096-token minimum and is processed uncached. On that prefix, Sonnet 5’s cache reads at $0.20 per million tokens cost less than Haiku 4.5’s uncached input at $1. Compare each model’s cost at your real cached share before acting on any per-token saving.
• Keep response caching out of evaluation. A cached response is evidence about the cache, not the model. rightmodeler leaves out of its evidence any replayed response that a gateway reports as a cache hit, such as a Portkey cache-status header or a Bifrost cache flag, and its gateway setup keeps response caching off on the replay route. Fallbacks, aliases and rewritten requests contaminate an evaluation the same way; how an AI gateway can invalidate your model evaluation covers all four.
• Fine-tune last. Training is the most expensive experiment, so run it after substitution has shown that no existing model clears the bar at an acceptable price.

Match the approach to the problem you can see in your traces:

• Frontier prices on routine steps: model substitution, proven per step on held-out cases.
• One step with easy and hard requests: routing, proven on held-out traffic.
• Long repeated prefixes: prompt caching, proven by the cached share of input.
• Exact repeated requests: response caching, proven by the duplicate rate.
• Oversized inputs: compression, proven by task success and cost net of cache effects.
• Latency-tolerant bulk work: batch APIs, proven by completion times.
• A narrow, high-volume task: fine-tuning or distillation, proven on held-out inputs at full lifecycle cost.

If the first line describes your bill, start with the traces you already have. How it works explains the replay-and-judge loop, and the B:Side Assist case study shows the evidence from one audit, workload by workload.