The context window is the new battleground: we now feed AI 19 tokens for every 1 it writes back

Everyone in AI worries about the same thing: the model writing too much, rambling, burning output tokens you have to pay for. When we sit down with the production data from our gateway, that is not the story at all. The output is stable. The input is what exploded.

I want to walk through what we see, because it changes how you should think about cost, caching, and where the next round of optimization work lives.

The ratio nobody is watching

Here is the number that stopped me. For every 1 token the model writes back, we now send it roughly 19 tokens of context on the way in. A year ago that ratio was closer to 10 to 1.

The answers did not get shorter. Models are as chatty as they were last year. What changed is how much we stuff into the prompt before the model ever starts generating. System prompts got longer. Tool definitions got heavier. Retrieval pipelines got more aggressive. Agents replay their entire history on every turn. All of that lands on the input side of the ledger.

To be careful about it, this ratio excludes the single largest organization each month, so one heavy customer is not dragging the line. This is the shape of the median workload, not an outlier.

The heavy tail is heavier than you think

Averages hide the interesting part. When we look at the heaviest 1% of prompts, the growth is even sharper.

The top 1% of requests now carry more than 258,000 tokens of input. That is not a chat message. That is an entire codebase, a full document set, or a long agent transcript being replayed into a single call. A year ago the same slice was well under half that size.

The typical request tells the same story in gentler terms. The average prompt keeps climbing quarter over quarter as teams lean on bigger context windows instead of tighter retrieval.

Why this matters for your bill and your latency

Two practical consequences fall out of this.

First, cost. If input is 19x your output and input keeps growing, then your marginal cost is dominated by tokens the model reads, not tokens it writes. The single highest-leverage optimization is no longer "make the model answer shorter". It is "stop paying full price to re-read the same context on every turn". That is what prompt caching is for, and it is why cache hit rate is now one of the first metrics we look at for any high-volume workload. We wrote up how automatic prefix caching works in the auto-caching docs.

Second, latency. Big prompts mean the model spends more time reading before it emits a first token. If you are running an interactive product, that shows up as a slower time to first token, and the fix is usually a combination of caching the stable prefix and being disciplined about what you put in the window.

What I would do about it

If you run agents or retrieval-heavy apps, audit your input side this quarter. Concretely:

• Measure your own input to output ratio. If it is climbing, that is your cost curve, not the output length.
• Turn on prompt caching for the stable parts of your prompt: system instructions, tool schemas, few-shot examples. See the caching guide.
• Attach request metadata so you can slice cost by feature and find the prompt that quietly tripled.
• Watch cost in one place with usage analytics instead of reconstructing it from provider invoices.

We publish the underlying operational cuts on our open data hub. If you want to see the per-provider numbers behind posts like this one, they live at requesty.ai/data.

The context window stopped being a spec sheet number and became the main cost driver in production AI. The teams that win the next year are the ones who treat the prompt as the product, not an afterthought.

Want the gateway to handle caching, routing, and cost tracking for you across 300+ models? Start free or read the quickstart.