How to Reduce LLM API Costs: 12 Proven Techniques

The fastest way to reduce LLM API costs is to send fewer tokens, to cheaper models, less often. Almost every technique below is a variation of one of those three levers: shrink input and output tokens, route work to the smallest model that does the job, and avoid calling the model at all when a cached or deterministic answer exists. Measure first, then apply the techniques in order of effort.

Why LLM API bills grow so quickly

Most providers charge per token, usually with separate rates for input and output tokens, and output tokens typically cost more. A request's cost is roughly:

cost = input_tokens * input_rate + output_tokens * output_rate

Costs balloon for predictable reasons: long system prompts repeated on every call, whole documents stuffed into context, chat histories that grow without bound, verbose answers, and using a flagship model for tasks a small model handles fine. If tokens are new to you, start with our tokenization guide and what is a large language model.

Step zero: monitor token usage

You cannot optimize what you cannot see. Before changing anything, log these per request:

• Feature or endpoint name
• Model used
• Input tokens, cached input tokens, and output tokens
• Latency and whether the result was served from cache

Aggregate by feature. In most apps, a small number of endpoints account for most of the spend, and that tells you where to start. This is technique 12 on the list, but you should do it first.

12 techniques to reduce LLM API costs

1. Cache exact responses

If the same input produces an acceptable same output, store it. A key-value cache keyed on a hash of the model, parameters, and prompt eliminates repeat calls entirely. This works well for FAQ answers, product descriptions, and classification of recurring strings. The usual caching rules apply: pick a TTL, and plan for cache invalidation when source data changes.

import hashlib, json

def cache_key(model: str, messages: list[dict], temperature: float) -> str:
    payload = json.dumps(
        {"m": model, "msgs": messages, "t": temperature},
        sort_keys=True,
    )
    return "llm:" + hashlib.sha256(payload.encode()).hexdigest()

def cached_complete(redis, client, model, messages, temperature=0.0, ttl=86400):
    key = cache_key(model, messages, temperature)
    hit = redis.get(key)
    if hit is not None:
        return hit.decode()
    text = client.complete(model=model, messages=messages, temperature=temperature)
    redis.setex(key, ttl, text)
    return text

Exact caching is most effective at low temperature, where the same prompt should give the same kind of answer.

2. Use provider prompt caching

Many providers now cache a long, stable prompt prefix on their side and bill those cached input tokens at a reduced rate. To benefit, structure prompts so the unchanging parts come first: system instructions, tool definitions, few-shot examples, and reference documents. Put the variable user input at the end. Reordering alone can make a large share of your input tokens cacheable.

3. Route requests to the right model

Not every request needs your most capable model. Build a simple router: send classification, extraction, and short rewrites to a small, cheap model, and reserve the large model for complex reasoning. A router can be rule-based (by endpoint), or a small classifier that estimates difficulty. A cascade is another variant: try the small model first, validate the output, and escalate only on failure.

4. Use batch APIs for offline work

If a job does not need an immediate answer, such as nightly summarization, backfilling embeddings, or bulk tagging, use a provider's batch endpoint where available. Batch processing is commonly offered at a discount in exchange for slower turnaround. The trade-off is the same one covered in batch vs streaming.

5. Set max tokens deliberately

Output tokens are usually the expensive ones. Set max_tokens per feature based on what the UI actually shows. A tooltip does not need a 1,000-token answer. Pair the limit with an instruction like "Answer in at most three sentences" so the model plans a short answer instead of getting cut off mid-sentence.

6. Shorten your prompts

Audit your system prompts. They often accumulate redundant rules, repeated examples, and polite padding. Remove instructions the model already follows by default, merge duplicates, and trim few-shot examples to the minimum that preserves quality on your evaluation set. For chat, summarize or truncate old turns instead of resending the full history forever.

7. Tune RAG top-k and chunk size

Retrieval-augmented generation often sends far more context than needed. If you retrieve ten chunks when the answer is usually in the top two or three, you pay for the rest on every call. Tune top-k, use a reranker to keep only the most relevant chunks, and right-size chunks so each carries one coherent idea. Our article on RAG explained covers retrieval tuning in more depth.

8. Ask for structured output

Free-form prose invites the model to explain itself. When code consumes the answer, request a compact JSON schema instead. Structured output reduces output tokens, removes preambles like "Sure, here is...", and makes parsing reliable, which also cuts retries caused by unparseable responses.

9. Add semantic caching

Users ask the same question in different words. A semantic cache embeds the incoming query, searches previous queries by vector similarity, and returns the stored answer when similarity exceeds a threshold. Set that threshold conservatively and scope the cache per tenant or context, since two similar-looking questions can need different answers. See embeddings explained for how similarity works.

10. Use smaller or specialized models

Beyond routing, consider whether a feature needs a general LLM at all. Embedding similarity, a fine-tuned small model, or even a regex can handle many classification and extraction tasks at a fraction of the cost. For high, steady volume, a self-hosted open-weight model may be cheaper than per-token pricing, though you take on infrastructure and operations work.

11. Stream responses to improve perceived speed

Streaming does not lower the token bill directly, but it changes the UX math. When users see text appear immediately, you can often use a cheaper, slightly slower model without hurting satisfaction. Streaming also lets users stop a generation they do not need, and stopping early saves output tokens with most providers.

12. Monitor, budget, and alert

Turn the logging from step zero into dashboards and alerts. Track cost per feature, cost per active user, and cache hit rate. Add per-user rate limiting and daily budgets so a runaway loop or abusive client cannot drain your account overnight.

Summary: which cost technique to use when

A practical order of operations

1. Instrument token usage per feature.
2. Apply the low-effort wins: max tokens, structured output, prompt reordering for prompt caching, exact caching.
3. Trim prompts and tune retrieval against an evaluation set so you can prove quality held.
4. Introduce model routing and semantic caching for the highest-spend endpoints.
5. Move offline workloads to batch and revisit self-hosting only if volume justifies it.

Key takeaways

• Every technique pulls one of three levers: fewer tokens, cheaper models, or fewer calls.
• Measure cost per feature before optimizing; spend is usually concentrated.
• Put stable content first in prompts to benefit from provider prompt caching.
• Route easy tasks to small models and escalate only when needed.
• Guard quality with an evaluation set so savings do not silently degrade answers.
• Budgets and rate limits protect you from runaway costs.

Frequently asked questions

What is the easiest way to reduce LLM API costs?

Start with low-effort changes: set sensible max tokens, request structured output, and cache repeated responses. Reordering prompts so the stable prefix comes first can also unlock provider prompt caching with no quality impact.

Is semantic caching safe?

It is safe when the similarity threshold is strict and the cache is scoped correctly, for example per tenant or per document set. Avoid it for answers that depend on user-specific or fast-changing data unless the cache key includes that context.

Does streaming reduce token costs?

Not directly, since you pay for the same tokens. It improves perceived latency, which can let you use a cheaper model, and it lets users cancel early, which avoids paying for output they did not need.

Should I self-host an open-weight model to save money?

Only if your volume is high and steady enough to keep GPUs busy, and your team can operate the infrastructure. For spiky or modest traffic, per-token APIs combined with routing and caching are usually simpler and cheaper overall.