AI Agent Cost Optimization: How to Cut LLM Spend by 80% with Routing

Inference is not just the biggest line item on your AI bill. In early 2026, Anthropic's engineering teams found that inference consumes over 85% of total enterprise AI budgets. The culprit is not cost per token, which has dropped steadily. It is the sheer volume of tokens that agentic workflows generate.

A single chatbot interaction might use 2,000 to 4,000 tokens. A single agent task with tool calls, planning steps, and verification loops? 50,000 to 500,000 tokens. Multiply that by hundreds of tasks per day, and you have a cost problem that scales linearly with adoption.

This guide covers the four techniques that production teams use to cut agent costs by 60 to 80 percent without sacrificing quality. If you are building the loops that drive this token volume in the first place, start with our companion guide on loop engineering.

Technique 1: Model routing (60 to 80% cost reduction)

Not every step in an agent workflow needs a frontier model. The insight is simple: route each step to the cheapest model that can handle it.

The 70/20/10 distribution

Production agent deployments in 2026 typically follow this pattern:

The math: if your unoptimized agent sends everything to Claude Opus 4.8 at $15/M input tokens, routing 70% to Flash ($0.30/M) and 20% to Sonnet ($3/M) drops your average cost from $15 to roughly $2.10 per million tokens. That is an 86% reduction. For the full breakdown of how this selection works per request, see smart routing demystified and how to route LLM requests by cost and latency.

Setting up routing with Requesty

from openai import OpenAI
 
client = OpenAI(
    base_url="https://router.requesty.ai/v1",
    api_key="your-requesty-key"
)
 
# Reference a named routing policy instead of a single model
response = client.chat.completions.create(
    model="policy/agent-cost-optimizer",  # the policy decides which model handles this call
    messages=[{"role": "user", "content": task_prompt}]
)

You define that policy once and reference it by name, so strategy changes never require a redeploy:

# requesty-policy.yaml
name: agent-cost-optimizer
routes:
  - name: fast-cheap
    match_intent: ["classify", "extract", "filter", "parse"]
    model: google/gemini-2.5-flash
 
  - name: balanced
    match_intent: ["summarize", "draft", "generate", "refactor"]
    model: anthropic/claude-sonnet-4.6
 
  - name: frontier
    match_intent: ["review", "architect", "debug-complex", "decide"]
    model: anthropic/claude-opus-4.8
 
  fallback:
    models:
      - anthropic/claude-sonnet-4.6
      - openai/gpt-5.4

For more policy patterns built specifically for agent workloads, see routing policies for agents.

Technique 2: Prompt caching (40 to 90% input cost reduction)

Agent workflows repeat the same prefixes constantly. System prompts, tool definitions, and RAG context can consume 40 to 60% of each request's token budget, and they are identical across calls.

How caching works

The provider stores the tokenized prefix. On subsequent requests with the same prefix, the cached portion costs a fraction of the full price:

Real-world impact

A support agent with a 4,000-token system prompt running 10,000 calls/day at Sonnet 4.6 pricing:

• Without caching: 4,000 tokens x 10,000 calls x $3/M = $120/day on system prompt alone
• With caching (90% hit rate): $12/day on system prompt
• Savings: $108/day, $3,240/month from one configuration

Requesty enables prompt caching automatically across all supported providers. No code changes required. We walk through the full caching math in how prompt caching cuts costs by up to 90%.

Technique 3: Context window management (30 to 60% savings)

The second biggest cost driver after repeated prompts is bloated context. Agents accumulate conversation history, tool results, and document chunks that grow unbounded.

Fixed retrieval budgets

Instead of retrieving a variable number of documents, enforce a strict token budget:

# Bad: unbounded retrieval
docs = retriever.get_relevant(query)  # might return 20K tokens
 
# Good: fixed budget
docs = retriever.get_relevant(query, max_tokens=4000)  # always 4K

Compaction

For long-running agent sessions, compress the conversation history periodically:

# After every 10 tool calls, compact the history
if len(messages) > 20:
    summary = summarize(messages[:-4])  # keep last 4 messages verbatim
    messages = [system_prompt, summary] + messages[-4:]

Claude Code and Codex both support automatic compaction. The loop controller summarizes completed work and resets, preventing context from growing indefinitely.

Hierarchical memory

Use a tiered memory system:

• Hot memory (in context): current task, last 3 steps
• Warm memory (file-based): today's completed work, available via tool call
• Cold memory (database): historical patterns, retrieved only when relevant

Technique 4: Budget caps and spend alerts

Cost optimization without hard limits is incomplete. A single runaway loop can erase a month of savings in an hour.

Per-agent budget caps

# Requesty budget configuration
budgets:
  - agent: pr-reviewer
    daily_limit: $10
    per_task_limit: $2
    alert_thresholds: [50%, 80%, 100%]
    on_exceed: pause_and_alert
 
  - agent: code-migration
    daily_limit: $50
    per_task_limit: $5
    alert_thresholds: [50%, 80%, 100%]
    on_exceed: pause_and_alert

Team chargeback

Track costs by team, project, and agent. Requesty breaks down spend by:

• API key (maps to team or project)
• Model used (shows routing effectiveness)
• Cache hit rate (shows caching effectiveness)
• Per-request cost (identifies expensive outliers)

Spend alerts

Configure webhook notifications when budgets hit thresholds:

{
  "alert_type": "spend_threshold",
  "threshold_percent": 80,
  "webhook_url": "https://hooks.slack.com/your-channel",
  "message": "Agent '{agent_name}' has used 80% of daily budget (${spent}/${limit})"
}

For the full setup, see budget caps and spend alerts and alerts when your LLM spend spikes.

Putting it all together: a real cost breakdown

Here is what a production coding agent workflow looks like before and after optimization:

The total monthly savings: $17,400. And this is for a single workflow. Teams running multiple agent loops see proportionally larger savings. The same gateway economics apply at the platform level, as we cover in how LLM gateways slash AI spend by up to 80%.

Getting started with Requesty

The fastest path to cost optimization:

1. Point your agents at Requesty. Change base_url to https://router.requesty.ai/v1. One line of code.
2. Define a routing policy. Set up a named policy with your model tiers and fallbacks, then reference it from your code with model="policy/your-policy".
3. Caching is automatic. Requesty enables provider-native caching on all supported models.
4. Set budget alerts. Configure daily limits and Slack notifications in the dashboard.
5. Monitor and tune. Use the cost analytics dashboard to identify expensive patterns and adjust routing.

The LLM gateway market is projected to reach $11 billion by 2030. The reason is simple: as agent adoption scales from 42% to near-universal in enterprise, cost optimization is not optional. It is the difference between a viable AI program and one that gets killed at the next budget review.

Start routing today. Your future self will thank you when the monthly invoice arrives.