Prompt Caching vs Semantic Caching for LLMs: Which Cuts Costs? (2026)

As of October 2026, OpenAI charges $0.10 per million cached input tokens for GPT-6.1 Sol, versus $2 for fresh input: 95% less. Prompt caching reuses a matching prefix and still generates an answer; semantic caching returns a stored answer for a similar request and skips the model call. Start with prompt caching, then add semantic caching where answers are safe to reuse.

Prompt caching vs semantic caching: the difference

The word "cache" hides three decisions. One reuses computation; the other two replay an answer. Pick exact response caching before semantic caching when identical requests recur: it avoids approximate matching, though it still needs freshness and access controls.

OpenAI's prompt caching guide describes prefix reuse. Redis LangCache describes response reuse on similar questions. Both can reduce latency, but a response-cache lookup adds work to misses.

How prompt caching works

When a model reads a prompt, it calculates intermediate key-value states, or KV states, for its tokens. A provider can save those states for a prefix and reuse them when a later request has the same eligible prefix. The provider still processes the new input and generates a response. OpenAI describes the stored KV tensors and prefix matching; DeepSeek notes that generated output can still vary.

Put the stable material first: tool definitions in a consistent order, system instructions, examples and shared documents. Put the current question and changing data after them. If a timestamp or reordered tool definition appears early, the prefix after that change may no longer match. Provider boundaries, eligibility and cache lifetimes also affect reuse.

Check where the provider stores cached state and which accounts or regions can reuse it. For a shared self-hosted vLLM deployment, use a trusted per-group cache_salt when cache isolation is required; vLLM documents salting to limit reuse between trust groups and protect against timing-based inference.

A breakpoint marks where an eligible prefix ends. In Anthropic's example, putting the breakpoint after a changing timestamp causes repeated writes and misses when no earlier entry was written. Place it after the last stable block. Concurrent requests can also miss before the first response starts and makes the entry available.

Providers differ on writes and retention

As of October 2026, OpenAI enables prompt caching by default on supported models. For GPT-5.6 and later, the minimum is 1,024 visible input tokens and the default 30-minute write rate is 1.25 times ordinary input. Anthropic supports automatic and explicit breakpoints; its default lifetime is five minutes, with an optional one-hour lifetime. Anthropic's minimum depends on the model.

Google's Gemini API enables implicit caching by default for Gemini 2.5 and newer models. DeepSeek enables disk context caching by default. Amazon Bedrock varies support by model and API. A gateway route can have different behavior and prices from a provider's direct API.

Now, the less welcome part: writing to the cache can cost more than sending fresh input. The write rate is the rate charged for those written tokens, not an extra fee added to the ordinary input rate. A write that is never read can increase the bill. See our provider cache-read price comparison for a broader price check.

How to cache LLM responses

A response cache sits in front of the model. On a hit, your application returns the stored answer. On a miss, it calls the model and stores an eligible answer for later. This is cache-aside: the application controls the lookup, the miss path and what gets saved.

Exact match first

Define an exact key from the request and the state that determines its answer. Include the model and generation settings where they affect the output contract. Include trusted authorization scope, prompt and schema versions, and any source version that affects correctness.

Cloudflare AI Gateway hashes the provider, endpoint, model, provider authentication header and full request body by default. Its cache is disabled by default. A custom key can group requests, so the application must preserve the authorization and response boundaries that the custom key removes.

Exact matching avoids a false match between two different keyed requests. It does not see an account balance or document change that was left out of the key. A time to live (TTL) limits the age of an entry; it does not guarantee the answer stays valid throughout that period. Purge or version entries when their sources or permissions change.

Semantic match second

A semantic cache searches for a question close enough to a stored one. Redis LangCache generates embeddings, searches for similar prompts and returns a stored response. An embedding is a numerical representation of text used for similarity search. Other designs use different methods: Cloudflare AI Search uses MinHash and locality-sensitive hashing.

Suppose a documentation assistant has answered "How do I reset my password?" A request for "Where do I reset my password?" could reuse the answer if both concern the same product, language, documentation version and reset workflow. "How do I reset my admin password?" needs a separate check. So does "How do I prevent password resets?"

A similarity score tells you two prompts look alike. It says nothing about whether the stored answer is safe to reuse. The vCache research paper explains why a fixed threshold alone gives no formal correctness guarantee. Test a threshold against your own requests and accepted answers.

A LangCache integration pattern

For a documentation assistant, create a LangCache service and configure tenant, locale and kb_version as attributes before using them. Derive the tenant from authenticated server-side identity. Add other answer dependencies to the scope or use separate caches; Redis limits configured attributes, so a composite version or separate cache can carry additional boundaries.

Set a finite service TTL based on the documentation's update cadence. Redis Cloud's default is no expiration. LangCache limits search and store prompts to 1,024 characters. Bypass this cache for longer questions; truncation can remove answer-relevant text.

This LangCache SDK pattern uses values supplied by your application. Install the Python SDK with pip install langcache. The threshold of 0.9 is an example to test, not a safety guarantee.

from langcache import LangCache
from langcache.models import SearchStrategy
 
cache = LangCache(server_url=HOST, cache_id=CACHE_ID, api_key=API_KEY)
scope = {"tenant": trusted_tenant_id, "locale": "en", "kb_version": "v7"}
 
candidate = cache.search(
    prompt=question,
    attributes=scope,
    search_strategies=[SearchStrategy.EXACT, SearchStrategy.SEMANTIC],
    similarity_threshold=0.9,
)

LangCache orders search strategies by priority. Its exact strategy compares prompt text case insensitively; the attributes still carry the other scope boundaries.

Before calling the model, inspect the candidate and return it only if it meets your acceptance and freshness rules. On a miss or rejected candidate, call the model. If the cache times out or fails, bypass it and call the model after the normal authorization checks. Set a lookup deadline and monitor cache errors. After checking that the fresh answer is safe to reuse, store it with the same scope:

cache.set(prompt=question, response=answer, attributes=scope)

When the knowledge base changes, use a new version in the scope and delete entries for the old version with delete_query(attributes=...), or delete individual entries with delete_by_id. Include the tenant in a deletion filter when the change applies to only one tenant.

Which workloads are safe to cache?

For response caches, derive tenant and user or permission scope from trusted authentication before lookup. Bind the feature, language, model or provider where relevant, prompt version, schema, policy and source version. Use access controls, retention and deletion for stored answers, which can contain sensitive data. Validate entries before writing them to reduce the risk of poisoning the cache.

Be careful with requests that trigger a tool call or side effect: replaying a response can suppress execution or repeat an old tool call. LiteLLM warns that semantic caching on multi-turn agent traffic can replay stale responses. An isolated read-only substep is a better candidate when every dependency is represented in its scope and validated.

How much money do the caches save?

The question nobody asks is how much semantic caching saves once prompt caching has already made each model call cheaper. Let's run the numbers.

Assume 1,000 requests, each with a 10,000-token stable prefix, a 1,000-token fresh suffix and 500 output tokens. Use OpenAI's direct GPT-6.1 Sol Standard rates as of October 2026: $2 per million fresh input tokens, $2.50 per million 30-minute cache-write tokens, $0.10 per million cache-read tokens and $10 per million output tokens.

For the prompt-only and combined cases, assume OpenAI Responses uses prompt_cache_options.mode: explicit with one breakpoint at the end of the stable prefix, leaving the suffix uncached. For the fresh-input and semantic-only cases, assume explicit mode with no breakpoints, so prompt caching is off. OpenAI's Responses SDK definition describes both settings; supported models otherwise use an implicit breakpoint by default.

For prompt caching, assume 50 whole-prefix writes and 950 whole-prefix reads. For semantic caching, assume 200 valid response hits and $0.10 in total lookup and storage costs. Assume no additional validation or operating costs beyond that $0.10. In the combined case, assume those hits are distributed independently of cache warmth: the 800 model calls have 40 writes and 760 reads. Exclude gateway fees and other fixed costs.

The semantic cache saves $1.74 on top of prompt caching: ($9.20 - $7.46) / $9.20, or about 18.9%. It does not save another 20% of the original $27 bill, because the calls it avoids were already cheaper with prompt caching. The result changes if hits are concentrated on expensive requests or if lookup and validation cost more.

For identical-sized prefixes where every miss writes the full prefix, the read fraction needed to beat fresh input is (write rate - fresh rate) / (write rate - read rate). At these rates, break-even is $0.50 / $2.40, or about 20.83% reads. A higher read fraction saves money on that prefix. Count output, other input, infrastructure and fees separately. For a wider comparison, calculate cost per completed task, including any extra review or retries.

Tools for prompt and response caching

We make Requesty, one of the products in this comparison. Pick Redis LangCache for an application-side semantic cache, Bifrost if you want to run an open-source gateway, and Requesty for provider prompt caching across supported routes. The table reflects documentation as of October 2026.

Portkey's semantic cache ignores the system prompt when matching: version it in a hard scope or purge entries when it changes. Portkey also documents a seven-day fallback when max_age is set below 60 seconds.

LiteLLM defaults to key, team and organization scope. Its semantic_cache_scope: end_user setting adds end-user isolation, but a missing end-user ID falls back to the wider scope. Enforce trusted identity and bypass the cache when it is absent.

Cloudflare's AI Search similarity cache is separate from AI Gateway's exact response cache. It also links entries to source chunks for invalidation when chunks change or are deleted.

Set up provider caching with Requesty

Requesty adds provider prompt caching and cache analytics behind one API. For providers that need explicit breakpoints, Auto Caching adds them to the largest content blocks. OpenAI, DeepSeek and Gemini implicit caching are handled automatically by those providers. The full request still reaches the model.

1. 1
   Create a key and prepare shared context

   Sign up for Requesty, create an API key, and top up your prepaid balance for the paid Claude Sonnet 5.5 model. Set REQUESTY_API_KEY in your environment and install the OpenAI Python SDK with pip install openai. Save a stable prefix in product_docs.txt; the Anthropic minimum for this model is 512 tokens.

2. 2
   Send requests with shared context

   Save the example as cache_example.py and run python cache_example.py twice within the provider's cache lifetime, waiting for the first response before the second call. For a third call, change only the user question and keep the context file unchanged; inspect response.usage for cached input.

   Python

   Copy

   import os
   from openai import OpenAI
    
   client = OpenAI(
       api_key=os.environ["REQUESTY_API_KEY"],
       base_url="https://router.requesty.ai/v1",
   )
    
   with open("product_docs.txt", encoding="utf-8") as f:
       static_context = f.read()
    
   response = client.chat.completions.create(
       model="anthropic/claude-sonnet-5-5",
       messages=[
           {"role": "system", "content": static_context},
           {"role": "user", "content": "How do I reset my password?"},
       ],
       extra_body={"requesty": {"auto_cache": True}},
   )
   print(response.choices[0].message.content)
   print(response.usage)

   The first eligible request can write the prefix; later calls can read it. If the request has several large blocks, check which prefix is being reused.

3. 3
   Inspect the result

   Open Requesty analytics. The Savings and Advanced views show cache metrics and let you group usage by model, provider, user or custom metadata. Compare cost as well as cached tokens.

To add response caching, put a separately chosen cache in your application before Requesty. Return an accepted hit there; send a miss to Requesty with provider prompt caching enabled. Instrument the application edge because response hits never reach Requesty.

Measure hits, quality and net cost

A request hit and a token hit answer different questions. Use these denominators for a consistent report:

Use comparable workloads for the baseline and cached periods, or estimate the baseline per request. Response hits can change cache warmth and therefore the cost of later model calls. A call with one cached token counts toward the prompt request hit rate, but its cached input share is small. Segment by workload and provider or model; compare end-to-end P50 and P95 latency on both hits and misses. If you sample answers for review, report the sample size and sampling method with the false-hit rate. Our guide to LLM observability metrics covers the wider measurement pipeline.

Usage fields differ by API. In the native Anthropic API, total input is input_tokens + cache_creation_input_tokens + cache_read_input_tokens. In OpenAI Responses, input_tokens already includes the breakdown in input_tokens_details, including cached_tokens and cache_write_tokens. Do not add those breakdowns to the total again. Requesty's Chat Completions schema names its write field caching_tokens.

Test semantic hits before serving them

Use a shadow test: run the cache lookup but return the fresh model answer. Review the candidate cached answer against the new request and current authorized source, not against how similar it looks to the fresh model's wording. Include near-neighbors with negation, product swaps, changed permissions, languages, dates and policy versions. Report adversarial near-neighbor results separately from the representative sample, with reviewed hit counts for each. Shadow lookups have their own cost.

Start with provider prompt caching: the model still answers each request, and Requesty handles the caching setup on supported routes. For a repetitive FAQ endpoint, add semantic caching only after the shadow test meets your error target. This week, check cached tokens by model in Requesty usage analytics, then use the API-specific input accounting above to calculate cached input share.