What Is an LLM Gateway, and Do You Need One? (2026)

A large language model (LLM) gateway is a service between your application and AI model providers. It gives your application a common API and can apply routing, fallback, spend controls and access policies to requests. You need one when multiple providers, teams or governance requirements make direct integrations hard to manage. A single-provider prototype usually does not need one.

How an LLM gateway works

An application sends a prompt and a model name to the gateway instead of calling the model provider directly. The gateway authenticates the request, applies its configured policies and forwards it to an allowed destination. The provider runs the model, and the gateway returns the response.

Picture a customer-support application. It sends a request using its gateway key and a policy that names a primary model and an approved backup. If the primary returns an eligible error, the gateway tries the backup according to that policy. If every target fails, the application receives an error. Requesty's fallback policy documentation describes this behavior.

The backup could be the same model hosted by a different provider, or a different model altogether. The second option changes what you have to test. Even with the same request shape, targets differ in context size, tool calling, output quality or data handling.

What an LLM gateway does

Gateways combine the following jobs; products differ in which ones they provide.

Routing and fallback solve different problems. A weighted policy distributes traffic across chosen targets. A latency policy selects among candidates using performance measurements. A fallback policy tries another target after an eligible failure.

Caching comes in two forms that people often mix up. With Requesty auto caching, a provider reuses eligible prompt prefixes while still processing the new request. A response cache stores an earlier answer and may return it without a new model call. If you evaluate response caching, check that cached answers cannot cross customer or permission boundaries.

Gateway logs cover traffic sent through the gateway. Your application also needs traces for retrieval, business logic and tool execution. Direct calls bypass the gateway's policies and logs, so include those paths in your inventory. To connect the two views, use a shared workflow identifier: Requesty's request metadata accepts a trace_id that you can also record in your application trace.

Do you need an LLM gateway?

No regulation requires a gateway. It is one place to enforce controls, provided every relevant application sends its requests through it. A provider's own controls or application-side instrumentation might be enough for a small deployment.

Now, the tradeoff: another service sits in the request path. It introduces a network hop, a dependency and potentially a fee. Measure time to first token and total request time both directly and through the gateway, using the same models, regions and workload. Keep a plan for an unavailable gateway as well as an unavailable provider.

The cost question

A gateway fee has to pay for itself: through a control you need, less operating work, or lower model spend. As of October 2026, Requesty adds a 5% fee on top of prepaid credit top-ups, with no seat fees or minimum spend.

Assume a workload costs $100 directly from a provider. Also assume the same model usage costs $100 in Requesty credits, you buy exactly the credits used, and there are no other charges. A $100 top-up costs $100 + ($100 x 0.05) = $105. If you select a different model that completes the same work to the same standard for $90 in credits, the top-up costs $90 + ($90 x 0.05) = $94.50. That is $5.50 less than the $100 direct baseline.

The assumption carrying that example is equal task quality. Compare cost per accepted result, including retries and review time, before switching models. Requesty does not automatically pick the cheapest model by cost.

Gateway, router or API gateway?

The product labels overlap a lot. Cloudflare's AI Gateway is an example of an API infrastructure product with AI-specific features. For definitions of more terms, see our LLM gateway terminology glossary.

Build, self-host or use a managed gateway

You have three options. A direct integration with your own shared library gives you control and avoids another service, but your team builds and maintains the features it needs. An open-source gateway gives you an existing service to operate. A managed gateway puts that operation with a vendor, while you still own your policies, application behavior and vendor review.

As of October 2026, these are the places we would start. We make Requesty, one of the products in this table.

Pick LiteLLM or Bifrost if an open-source, self-hosted gateway is mandatory. Your team then owns deployment, upgrades, credential protection and availability. For a broader shortlist, see the 2026 AI gateway comparison.

Test the failure path before production

A fallback policy only helps if the backup can do the same job. Test with representative prompts and tool calls, then force failures before an outage occurs.

For a gateway that supports configurable upstream endpoints, point the test primary and backup at separate staging stubs. Configure the fallback policy to try the backup on an HTTP 429. Have the primary return a provider-compatible 429 response before the stream starts and the backup return a valid success response. Send a synthetic request with client retries disabled and a known gateway retry setting. Check both stub request journals and the client response to confirm the primary was attempted, the backup was called and its result was returned. Repeat with both targets failing, then test partial streams separately. WireMock's stubbing guide shows how to return canned HTTP responses.

For Requesty, the fallback documentation describes filtering Analytics by policy to see model successes, failures and fallback frequency. Use that view to review policy behavior alongside your application tests.

A support agent that has already displayed half an answer cannot safely pretend a new stream is a continuation of the original. It also needs to avoid issuing a second refund if a tool call was already executed. The OpenAI Python SDK states that stream consumption is not automatically retried because replay could duplicate output already delivered to the application.

Look at failure domains too. Two routes that use the same provider account, region or gateway do not provide independent protection against a failure of that shared dependency. Account for SDK retries too: the OpenAI Python SDK retries certain errors by default, so gateway retries can extend the total time spent on one user request.

Check both gateway and inference regions

Gateway location and model location are separate choices. Requesty's EU routing documentation places its EU gateway in Frankfurt. To keep Requesty processing in the EU, use https://router.eu.requesty.ai/v1 with the OpenAI SDK and restrict the organization's allowed Requesty regions to EU. For EU inference, select EU-hosted models and restrict approved targets, including fallbacks.

Review logs and provider terms alongside region selection. Requesty's data privacy documentation separates operational telemetry from payload logging. On Enterprise, Requesty's organization-wide zero data retention setting permanently prevents new payload logging. It does not change upstream provider retention or erase historical payload logs.

Requesty's guardrails scan request content for personal information and secrets, with report or mask actions. They do not scan model responses for newly generated sensitive information. A gateway control helps enforce a policy; it does not certify that the entire application complies with a regulation.

Try a gateway with the OpenAI SDK

Requesty gives your team a shared endpoint for 600+ models, with per-key spend limits, usage analytics and configurable fallback policies. Team groups are an Enterprise feature; a shared group budget requires Group Budget mode.

To use the OpenAI SDK with Requesty, set a Requesty key and gateway base URL, then select a compatible provider/model ID. The quickstart shows the endpoint change.

1. 1
   Create a key and install the SDK

   Sign up for Requesty, create an API key and top up your balance. The example uses a paid model. Set your key in your shell and install the OpenAI Python SDK:

   Shell

   Copy

   pip install openai
   export REQUESTY_API_KEY="your-requesty-key"

2. 2
   Send a request

   Save this as gateway.py and run python gateway.py:

   Python

   Copy

   import os
   from openai import OpenAI
    
   client = OpenAI(
       api_key=os.environ["REQUESTY_API_KEY"],
       base_url="https://router.requesty.ai/v1",
   )
    
   response = client.chat.completions.create(
       model="openai/gpt-6-sol",
       messages=[{"role": "user", "content": "Explain an LLM gateway in one sentence."}],
   )
   print(response.choices[0].message.content)

   If you restricted your organization to EU routing, change the base URL to https://router.eu.requesty.ai/v1 and select an approved EU-hosted model, such as bedrock/claude-sonnet-5-5@eu-central-1.

   The OpenAI-compatible base URL and request shape come from the Requesty quickstart. The model library lists available models.

3. 3
   Add a fallback if the workload needs one

   Create a fallback policy in Routing Policies, add compatible approved targets and save it as support-fallback. Then replace the model argument in the example with:

   Python

   Copy

   model="policy/support-fallback"

   The policy must exist before you call it. Follow the fallback setup guide, then run the failure tests above.

So what?

So when do you need one? Two signals say it is time: your application needs a tested backup across model providers, or your teams need shared spend and access controls. This week, pick one workload and set up its first control: a fallback policy for the former, or a separate key with a spend limit for the latter.