LLM Load Balancing Across API Keys and Providers (2026)

LLM load balancing puts a router between your application and approved model endpoints. It distributes requests by a rule such as weights or observed load. Add a fallback chain to handle eligible route failures. Multiple API keys add capacity only when they have independently provisioned quota. Keep conversations on the same route when cache reuse is useful.

Start with the quota, not the key

A route is more than a credential. It has a provider, account or project, model, region and set of limits. Two routes can look separate in your configuration while drawing from the same upstream allowance.

As of October 2026, the quota boundaries for three major APIs are:

Assume two approved endpoints each have an independent allowance of 100 requests per minute, with no higher shared cap. Their nominal combined allowance is 100 + 100 = 200 requests per minute. If both keys instead draw on one shared 100 request-per-minute parent limit, the combined allowance remains 100. Actual throughput also depends on token limits, bursts and available capacity.

Inventory requests per minute, input and output tokens per minute, daily limits, concurrency and spending caps for each route. A 50/50 split by request count is not a 50/50 split by tokens if one route receives longer prompts. Ask for more capacity or pace a queue when a shared limit is the constraint. Our provider rate limits guide covers the quota dimensions in more detail.

Use authorized credentials and review the provider agreement before pooling accounts. OpenAI's business terms prohibit transferring API keys with third parties and circumventing rate or usage limits.

Here, key rotation means distributing calls across credentials. Security rotation means replacing and revoking credentials; you need a process for both.

Choose a routing rule

Load balancing chooses a route while options are healthy. A fallback chain specifies what to try after an eligible failure. They work together: a weighted policy can select between two branches, each with its own fallback.

LiteLLM's routing documentation defines least busy using ongoing calls and describes usage-based selection. Router measurements are separate from the provider's authoritative limits. LiteLLM's simple-shuffle strategy uses weighted random selection, as shown in the version 1.106.2 implementation. It is not weighted round robin.

Only put routes in the same pool after checking context length, tool calls, structured output, streaming, data handling and task quality. A shared API format does not make two models interchangeable. For regional requirements, approve every primary and fallback inference endpoint. The gateway's region and the model's inference region are separate choices.

Three ways to balance LLM requests

We make Requesty. For a self-hosted proxy or a pool of multiple customer keys from the same provider, LiteLLM is the better fit for these examples. Requesty's BYOK documentation specifies at most one key per provider.

Rotate two OpenAI keys in one process

Install the OpenAI Python SDK with pip install openai. Set OPENAI_KEY_A and OPENAI_KEY_B to your authorized keys in the environment. These variable names belong to this example; the code passes them explicitly to the SDK.

import os
from itertools import cycle
from openai import OpenAI
 
clients = cycle([
    OpenAI(api_key=os.environ["OPENAI_KEY_A"], max_retries=0),
    OpenAI(api_key=os.environ["OPENAI_KEY_B"], max_retries=0),
])
 
for _ in range(2):
    response = next(clients).chat.completions.create(
        model="gpt-6-luna",
        messages=[{"role": "user", "content": "Say hello."}],
    )
    print(response.choices[0].message.content)

If both calls succeed, this sends the first to A and the second to B. The OpenAI model documentation lists gpt-6-luna as an API model ID; your account needs access to it. This example disables SDK retries to make each selection explicit. An error stops the loop.

The counter is local to one process. Separate workers do not coordinate, and the snippet has no health checks, pacing, fallback or session affinity. Add those mechanisms or move selection to a shared router when the workload needs them.

Run a self-hosted LiteLLM proxy

This example creates one public alias with two OpenAI keys and one Anthropic route. The weights are 35, 35 and 30, so the intended starting selection probabilities are 35%, 35% and 30% among eligible routes. Those are probabilities, not quota reservations. GPT-6 Luna and Claude Sonnet 5 are different models; evaluate both before sending production requests to the pool.

Set OPENAI_KEY_A, OPENAI_KEY_B, ANTHROPIC_API_KEY and LITELLM_MASTER_KEY in the proxy's environment. The master key must start with sk-, according to LiteLLM's virtual key documentation. The two OPENAI_KEY names are local names for this configuration.

Save the following as config.yaml. The model paths follow LiteLLM's OpenAI and Anthropic provider configuration, and weight sits inside litellm_params as in the routing documentation.

model_list:
  - model_name: support-pool
    litellm_params:
      model: openai/gpt-6-luna
      api_key: os.environ/OPENAI_KEY_A
      weight: 35
    model_info:
      id: openai-a
  - model_name: support-pool
    litellm_params:
      model: openai/gpt-6-luna
      api_key: os.environ/OPENAI_KEY_B
      weight: 35
    model_info:
      id: openai-b
  - model_name: support-pool
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
      weight: 30
    model_info:
      id: anthropic-a
router_settings:
  routing_strategy: simple-shuffle
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Install and start the proxy on loopback:

pip install 'litellm[proxy]==1.106.2'
litellm --config config.yaml --host 127.0.0.1

The example uses Python 3.10 through 3.14 and is pinned to LiteLLM 1.106.2. An October 21 security report identified compromised 1.107.0 wheels and listed the 1.106.2 wheels among those inspected for that incident. For production, review current advisories, pin a reviewed release and verify artifact provenance. Keep the proxy on a private network, use TLS, and issue scoped keys to applications instead of sharing the master key.

In another shell with the master key set, send a smoke test:

curl -i http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"support-pool","messages":[{"role":"user","content":"Say hello."}],"max_completion_tokens":1024}'

Inspect x-litellm-model-id and the response. LiteLLM documents this header as the deployment identifier. Repeat with independent requests to see selections; a short run will not reproduce exact percentages. The output cap is an upper bound, including any billable reasoning tokens.

Add and test a LiteLLM fallback

The configuration above is a weighted pool. To add a separate backup, set GEMINI_API_KEY and append this entry to the existing model_list. The model and credential syntax follow LiteLLM's Gemini guide. Check the backup's access, quota, data rules and task quality first.

  - model_name: support-backup
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
    model_info:
      id: gemini-backup

Replace the existing router_settings block with this one, keeping the original general_settings block:

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  fallbacks:
    - support-pool: [support-backup]

Save config.yaml, stop the original proxy with Ctrl+C, and restart it with litellm --config config.yaml --host 127.0.0.1 so the backup configuration takes effect.

The LiteLLM fallback guide describes moving to the backup group after the primary call and its retries fail. A retry can pick another deployment in the primary pool; it does not guarantee that every primary was tried. Regular fallbacks cover more than transient errors, so review which failures should reach a backup, including content-policy failures.

model_list:
  - model_name: support-pool
    litellm_params:
      model: openai/gpt-6-luna
      api_base: http://127.0.0.1:9999/v1
      api_key: sk-stage-only
      weight: 35
    model_info:
      id: stage-a
  - model_name: support-pool
    litellm_params:
      model: openai/gpt-6-luna
      api_base: http://127.0.0.1:9999/v1
      api_key: sk-stage-only
      weight: 35
    model_info:
      id: stage-b
  - model_name: support-pool
    litellm_params:
      model: openai/gpt-6-luna
      api_base: http://127.0.0.1:9999/v1
      api_key: sk-stage-only
      weight: 30
    model_info:
      id: stage-c
  - model_name: support-backup
    litellm_params:
      model: openai/gpt-6.1-terra
      api_base: http://127.0.0.1:9999/v1
      api_key: sk-stage-only
    model_info:
      id: stage-backup
router_settings:
  routing_strategy: simple-shuffle
  num_retries: 0
  fallbacks:
    - support-pool: [support-backup]
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

import json
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        model = body.get("model")
        backup = model == "gpt-6.1-terra"
        if backup and os.environ.get("FAIL_BACKUP") != "1":
            status = 200
            result = {
                "id": "chatcmpl-stage", "object": "chat.completion",
                "created": 1, "model": model,
                "choices": [{"index": 0, "finish_reason": "stop",
                             "message": {"role": "assistant", "content": "backup"}}],
                "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
            }
        else:
            status = int(os.environ.get("PRIMARY_STATUS", "429"))
            result = {"error": {"message": "staged upstream error",
                                "type": "rate_limit_error" if status == 429 else "server_error"}}
        data = json.dumps(result).encode()
        print(f"model={model} status={status}", flush=True)
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)
 
ThreadingHTTPServer(("127.0.0.1", 9999), Handler).serve_forever()

PRIMARY_STATUS=429 python stub.py

export LITELLM_MASTER_KEY=sk-stage-only-proxy-key
litellm --config stage.yaml --host 127.0.0.1

export LITELLM_MASTER_KEY=sk-stage-only-proxy-key
curl -i http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"support-pool","messages":[{"role":"user","content":"Say hello."}]}'

Stop the staging proxy and stub. In a shell with your original OPENAI_KEY_A, OPENAI_KEY_B, ANTHROPIC_API_KEY, GEMINI_API_KEY and secret LITELLM_MASTER_KEY set, restart the original proxy with litellm --config config.yaml --host 127.0.0.1. Do not use the staging master key for the original proxy.

For multiple proxy instances, LiteLLM documents Redis settings for shared routing state. Configured rpm and tpm values inform selection by default; the documented enforce_model_rate_limits check adds enforcement, with token enforcement best effort. Neither turns a shared provider quota into independent quotas.

Create a managed Requesty policy

Requesty's managed gateway lets you configure weighted provider and model routes without operating the proxy. You can use a Requesty-provided key or your own supported provider credential for each model. Requesty is free to get started; pay as you go adds a 5% fee on top of credit top-ups, with no seat fee or minimum spend. See pricing.

The following example uses a 70/30 split and an explicit conversation trace, following the load-balancing policy guide.

1. 1
   Create your Requesty key

   Sign up, create a Requesty API key, and add credits for the paid models in this example. To use your provider credentials, add them under Bring Your Own Keys. The BYOK documentation specifies at most one key per provider. The free tier covers free models; see Requesty pricing.

2. 2
   Create the weighted policy

   Open Routing Policies, select Create Policy, choose Load balance, and name it support-split. Add openai/gpt-6-luna at 70% and anthropic/claude-sonnet-5-5 at 30%. Choose Requesty provided key or My own key for each entry, then save. Check current availability in the Requesty model library and confirm both models meet your workload and data requirements.

   For example, the Claude Sonnet 5.5 API rejects temperature, while GPT-6 Luna supports Chat Completions function calling only with reasoning_effort set to none. Test the parameters and tool behavior your application sends to both routes.

3. 3
   Call the policy

   Set REQUESTY_API_KEY in your environment and use the OpenAI Python SDK:

   Python

   Copy

   import os
   from openai import OpenAI
    
   client = OpenAI(
       api_key=os.environ["REQUESTY_API_KEY"],
       base_url="https://router.requesty.ai/v1",
   )
   response = client.chat.completions.create(
       model="policy/support-split",
       messages=[{"role": "user", "content": "Say hello."}],
       extra_body={"requesty": {"trace_id": "session-123"}},
   )
   print(response.choices[0].message.content)

Use a distinct application-owned trace_id for each conversation and reuse it on later turns. Include the conversation history needed by your application. To measure distribution, send requests with many distinct traces and inspect the policy's Live Performance panel and analytics.

For failover, create fallback policies with approved backup models, then select those policies as the targets of a load-balancing policy. The example above selects individual models and does not configure that chain. If every route fails, return an error to the application.

For a speed-led policy instead, create a policy named support-fast, choose Latency, add compatible candidates, and call it with model="policy/support-fast". Requesty's latency routing considers response start time, generation speed and recent observations, penalizes recent errors, and tries candidates in speed order on failure. It does not produce a fixed percentage split or select by price. See our routing policies guide for more on composing policies.

Keep conversations on a useful route

OpenAI's caching guide requires a matching rendered prefix and an available cache entry on the selected machine. Its caches are not shared across organizations or regional processing boundaries. A second key in the same organization does not, by itself, establish a separate cache.

Put stable instructions and tool definitions before changing conversation content. For the Requesty Python example, enable provider-side prompt caching by adding "auto_cache": True alongside "trace_id" inside the requesty object in extra_body. Requesty's auto-caching guide explains how it adds breakpoints for providers that need them. Providers have minimum cacheable prefix lengths and different cache rules; check cache-read usage for each route. Auto caching uses provider-level prompt caching, not stored responses, so continue to send the required history.

A failure, policy change, expired entry or different eligible endpoint can change the result.

Affinity also changes the split you observe. Assume 100 sessions are assigned using 70/30 weights. One session assigned to the 70 side sends 1,000 requests, and each of the other 99 sessions sends one. That one session contributes 1,000 of 1,099 requests, about 91%. Even a 70/30 split of sessions would not yield a 70/30 split of requests. Measure tokens and spend too, since neither has to match the session split.

There is a correctness issue as well as a cache issue. LiteLLM's Responses API guidance explains that encrypted reasoning may be tied to the originating API key. Check its affinity option and your proxy version before moving a conversation with encrypted state between deployments.

Test the failure path

A successful hello request proves that one route answered. It does not prove that a backup is eligible or that your retry budget is bounded.

For a 429, inspect the response before changing weights. OpenAI's rate-limit guide distinguishes temporary throttling from quota or billing errors that need an account fix. If Retry-After is present and valid, wait at least that long for a retry to the affected route. If it exceeds your deadline, defer the request. Without a valid hint, use bounded exponential backoff with jitter for transient errors. Our 429 diagnosis guide covers the response details.

Check the origin when Requesty returns an error. Its error reference distinguishes router from provider. A provider 429 may be helped by an approved route with independent capacity. A Requesty in-flight concurrency 429 occurs before the request is routed: bound client concurrency and wait for a slot. Requesty does not queue that rejection or impose its own requests-per-minute limit.

Retries multiply when the SDK, gateway and application each have their own loop. Set an overall attempt and elapsed-time budget, including the SDK's automatic retries. Do not blindly replay a stream after output or a tool call has reached the client. A timeout also leaves open the possibility that the upstream processed the request; protect side effects from duplication.

Finally, a second provider behind one gateway does not protect against a gateway outage. If that failure mode is in scope, design and test an application-level alternate path with the same model, data, region and accounting controls.

Put the first route into production

Start with one workload and inventory its actual quota boundaries. Run the same test set against each eligible route, configure the split, and inject a failure before increasing traffic. For a managed implementation, create a Requesty load-balancing policy and add fallback policies for the errors your application needs to survive.