$1.8k before anyone noticed: how to cap runaway agent spend

Three posts from the same week, from three different communities, all describing the same failure:

A multi agent system that burned through about $1.8k before anyone noticed. A thread asking flatly how are y'all capping agent spend. An agency asking whether anyone running agents for clients knows what each client costs them. Add a team burning $100 a day in API overages and someone whose engineering key was stolen and used for 11 million tokens in minutes.

That is a category of problem, not five unlucky people. Agents changed the spend profile of LLM apps and most cost tooling has not caught up.

Why agent spend fails differently

A chat app spends money when a human types. The ceiling is human speed. An agent spends money when a loop iterates, and the ceiling is your timeout.

Four specific mechanics do most of the damage:

Retry storms. A tool fails, the agent retries with the failure appended to context. Context grows every attempt, so attempt seven costs several times attempt one. Nothing errors out. The bill compounds quietly.

Context accumulation. Long running sessions carry their whole history. Cost per step rises through the run even when the work does not.

Fan out. One task spawns five subagents, each spawning tools. A ten cent request becomes a three dollar tree, and the tree depth depends on the input.

Silent volume. None of the above trips a rate limit. The requests are valid and in quota. This is why one thread was specifically about detecting loops and retry storms that burn spend without triggering normal rate limits.

Add a leaked key and you have the same shape with an adversary instead of a bug.

The setup we recommend

Five layers, cheapest to hardest. Do the first two today.

1. A hard limit on every key

Not a budget in a config file your agent can ignore. A limit enforced where the request is served. Set spend limits, expiry, and rate ceilings per key with API limits and the key management API so provisioning is scriptable.

The rule we use: one key per surface. Production API, background workers, each internal tool, each customer if you are an agency, and one throwaway key per developer. A key that can spend your whole month is a key you cannot safely give to an agent.

2. Alerts on spend velocity

Daily totals arrive too late for something that can lose $1.8k in an afternoon. Alerts should fire on rate of change, not just thresholds, and go somewhere a human reads at 2am. The pattern is in alerts when your LLM spend spikes.

Useful triggers:

• spend in the last hour above 3x the trailing 24 hour hourly average
• a single session over a fixed dollar ceiling
• tokens per request above a per workload maximum, which catches context accumulation before it catches you
• error rate above a threshold, since a rising error rate plus rising spend is a retry storm in progress

3. Attribution good enough to answer "who did this"

Tag every call with team, customer, feature, and environment using request metadata, then read it back through cost tracking and usage analytics. For agency style multi tenant setups, use groups with per member budget overrides so one client cannot consume another client's headroom. The key naming discipline is in labeling API keys for cost attribution.

The test: when spend doubles, can you name the cause in under two minutes. If not, you are missing tags, not dashboards.

4. Loop and repetition detection

The signal is not request rate, it is repetition. Watch for the same tool call with the same arguments repeated inside one session, growing prompt size with a flat completion size, and step counts past a sane maximum. Logs and session reconstruction let you replay a run and see where it started circling, which is usually one bad tool response.

Then act on it in the harness: a hard step cap per task, a dollar cap per session checked before each step, and a rule that two identical failures means escalate or stop rather than retry a third time.

5. Make each step cheaper

Capping the downside is not the same as fixing the economics. Two levers move the base rate:

Caching. Agent traffic repeats a system prompt and tool schema on every step. Auto caching makes that a one time cost per prefix.

Right sizing the model. One engineer tracked 30 days of coding agent usage and found about 80 percent of tasks did not need the frontier model. That matches what we see: cheap models handle most steps, and the expensive model earns its price on a minority. Escalate on signal rather than defaulting high, which is the discussion in routing policies for agents and the numbers in AI agent cost optimization. Use approved models to keep an experimental branch from quietly switching everything to the most expensive option.

The escalation question

The sharpest version of this problem was asked on r/LLMDevs: what signal should trigger escalation to a stronger model instead of retrying. Retrying the same model on the same context is the most expensive way to fail, because you pay again for the same mistake with more tokens.

What works in practice:

• escalate on structured failure, such as invalid JSON or a schema violation, not on vague dissatisfaction
• escalate once, then stop, and let a human or a queue take it
• never escalate on timeout alone, since a timeout usually means the provider, and a fallback policy at the same tier is the cheaper answer
• log every escalation with its trigger so you can measure whether the expensive model changed the outcome

Do this today

If you only take two actions: put a spend limit on every key, and set an hourly velocity alert. Both are configuration, both take minutes, and together they turn a four figure incident into a page at 3 percent of the cost.

Start at API limits, then read budget caps and spend alerts. If you want to see per customer cost before you commit to anything, the quickstart is a base URL swap and tagged requests show up in analytics immediately.