Why AI gateways should be built in Go: a language audit of every major gateway

An AI gateway sits on the hottest path in your stack. Every prompt, every token of every streamed response, every retry after a provider 429 flows through it. If the gateway is slow, everything you build on top of it is slow. If it drops a stream mid-response, your agent loop breaks in ways that are miserable to debug.

So it is worth asking a question that gateway vendors rarely answer directly: what is this thing written in, and what does that runtime do under load?

We went through every major AI gateway we could find, checked the language and runtime each one runs on, and collected the public benchmarks. Then we looked at our own codebase, because Requesty's router is written in Go, and the longer we run it in production the more convinced we are that Go is the right default for this category. This post is the full argument: the landscape, the physics, the receipts from our own code, and an honest comparison with the most interesting counter-design out there, OpenRouter on Cloudflare's edge.

The landscape: every major AI gateway and what it runs on

Sources for the non-obvious rows: OpenRouter's own platform engineering job listings describe "edge and cloud infrastructure across Cloudflare, Google Cloud, and Vercel" and "a full-stack TypeScript shop" building on Cloudflare Workers. Vercel documents AI Gateway as a Node.js service on Fluid compute. Helicone explains its Rust rewrite in its launch post. LiteLLM announced its Rust gateway beta in July 2026, which is itself an admission worth pausing on: the most widely deployed Python gateway is rewriting its data plane in a compiled language.

Three clusters emerge:

1. Interpreted or JIT runtimes (Python, Node.js, TypeScript on V8): LiteLLM, Portkey, Vercel, OpenRouter.
2. Nginx-era proxies with scripting layers (Lua): Kong, APISIX.
3. Compiled, natively concurrent binaries (Go, Rust): Requesty, Bifrost, Helicone, TensorZero, LangDB, Envoy AI Gateway, Higress.

Nearly every gateway born after 2024 is in cluster 3, and two prominent members of clusters 1 and 2 (Helicone, LiteLLM) have migrated or are migrating into it. The direction of travel is one way.

What an AI gateway does, mechanically

Before arguing about languages, be precise about the workload. An AI gateway is not a web app. Per request it must:

• Authenticate an API key or JWT, in single-digit microseconds if possible.
• Fetch per-key configuration: routing policy, model allowlist, budgets, guardrails, BYOK credentials.
• Compile a routing plan: an ordered list of (provider, model) candidates from a policy.
• Open an upstream connection to a provider, ideally reusing a warm TLS connection from a pool.
• Stream the response token by token, flushing every chunk immediately, for anywhere from 200ms to 10+ minutes for long agentic turns.
• Fail over mid-flight when a provider returns 429/529, without ever double-sending bytes to the client.
• Do the bookkeeping afterwards: count tokens, compute cost, deduct balance, write telemetry, fire webhooks. None of this may block the response.

Two properties dominate: massive I/O concurrency with tiny CPU slices (thousands of open streams, each mostly waiting on a provider), and hot shared state (config caches, latency statistics, connection pools) that must be readable in nanoseconds by every in-flight request.

That profile is exactly what Go was designed for. Goroutines make a 10,000-concurrent-stream server boring: each stream is a cheap green thread, the scheduler multiplexes them over a few OS threads, and there is no async/await ceremony and no single-threaded event loop to stall. Shared state is an in-process map behind an RWMutex, not a network hop to Redis. The result is a single static binary that saturates a NIC before it saturates a CPU.

It is the same reason the previous generation of cloud data planes converged on Go: Docker, Kubernetes, Traefik, Caddy, Tyk, KrakenD. AI gateways are the same shape of software with a streaming-heavy twist.

The numbers: what the runtimes measure

Public benchmarks across the category, all measuring gateway processing overhead with the provider call mocked or excluded:

TensorZero's head-to-head is the starkest: their Rust gateway holds sub-millisecond p99 at 10,000 QPS on one instance, while LiteLLM's Python proxy degrades at hundreds of QPS and fails outright at 1,000. Maxim measured a Go gateway (Bifrost) against the same Python proxy at 500 RPS: 9.5x the throughput, 54x lower p99, one third the peak memory.

The lesson is not "Python is bad." Python is a fine control-plane language; half our own analytics tooling is Python. The lesson is that the data plane, the thing holding thousands of concurrent token streams, punishes interpreted runtimes and rewards compiled ones. The measured gap between Go and Rust is about a millisecond. The gap between either and Python is one to two orders of magnitude, and it widens under exactly the load spikes where reliability matters most.

In end-to-end production terms, where overhead includes auth, config fetch, policy evaluation, and accounting rather than a bare proxy hop, our measurements put Requesty at ~16ms of added latency per request, OpenRouter at ~55ms, and self-hosted LiteLLM at ~124ms.

The interesting counter-design: OpenRouter on Cloudflare's edge

OpenRouter is the most successful gateway not built on a compiled binary, so it deserves a serious treatment rather than a strawman.

Their architecture, as described in their own hiring materials, is TypeScript on Cloudflare Workers: requests land on Cloudflare's global edge, execute in V8 isolates, and are backed by Postgres, Spanner, and ClickHouse. The genuine advantages:

• First-hop proximity. Your request hits a point of presence tens of milliseconds away, anywhere on earth, with TLS terminated close to the user.
• Zero-ops scaling. Isolates spin up in microseconds; there are no pods to autoscale and no capacity planning.
• A great fit for TypeScript teams. One language across the dashboard, API, and edge logic.

But the serverless edge model collides with the mechanics of gateway work in three specific places:

1. No durable in-process state. A V8 isolate can be evicted at any moment and holds 128MB of memory. You cannot keep a hot routing-config cache, an hour of latency statistics, or a compiled policy table resident in memory across requests the way a long-lived process can. State lives a network hop away (KV, Durable Objects, Postgres), and that hop is paid on the request path. This is why OpenRouter's own docs describe key validation against a database on the hot path, and it is part of why the measured overhead gap exists.

2. No persistent upstream connection pools. A long-lived Go process holds warm TLS connections to OpenAI, Anthropic, and Bedrock, so a request pays zero handshake cost. Isolates cannot own sockets across invocations; connection reuse is best-effort at the platform layer, outside the gateway author's control.

3. Inference does not benefit from the edge. Edge computing wins when the work happens at the edge. But an LLM request spends 99% of its life inside a provider's datacenter in Virginia, Oregon, or Frankfurt. Terminating the request 20ms closer to the user, then hauling it to the same GPU cluster anyway, optimizes the smallest term in the latency equation, while giving up the warm state and pooled connections that optimize the terms the gateway controls: config lookup, policy evaluation, retry speed, and failover.

There is also a fourth, softer cost: on a proprietary edge runtime, you cannot hand an enterprise a binary. OpenRouter offers no self-hosted or in-VPC deployment path. A compiled gateway can run anywhere a container runs, which is what makes regional and on-prem deployment a configuration change rather than a re-architecture.

Helicone, which ran its original proxy on Cloudflare Workers for years, put the conclusion bluntly in its Rust rewrite announcement: "You wouldn't use NGINX written in TypeScript or Python. Critical infrastructure demands performance, reliability, and predictable behavior under load." They chose Rust; the reasoning applies equally to Go.

What we did in Go: the receipts

Claims about language choice are cheap, so here is what the choice looks like inside Requesty's router, concretely.

The hot path never waits on bookkeeping

The router is a Gin HTTP server with a strict rule: post-processing never blocks the response. Balance deduction, cost metrics, telemetry insights, and webhooks all fire in goroutines after the last byte reaches the client. In Go this is one keyword, not a task queue, not a lambda, not a callback pyramid. The client-visible latency budget is spent only on auth, config, and the provider call.

Config caching with singleflight

Per-key configuration (policies, allowlists, budgets, BYOK keys) is served from an in-process generic refetch cache with two TTLs: a stale TTL after which the value is refreshed in the background while requests keep being served the stale copy, and a dead TTL after which it must be refetched. Concurrent refreshes for the same key are collapsed with golang.org/x/sync/singleflight, so a traffic spike on one key produces exactly one upstream fetch instead of a thundering herd. This pattern needs a long-lived process with shared memory; it is impossible in an isolate that may not exist a second from now.

Typed failover that knows when it must not retry

Every provider error is translated into a typed RouterError carrying a ShouldFallback() flag. Rate-limited (429), overloaded (529), malformed upstream response, and timeouts fall through to the next provider in the plan. A client mistake (400) does not. And critically, once a stream has started and bytes have reached the client, the error becomes non-retryable, because silently replaying a partial stream corrupts agent state. Getting this right requires owning the response writer through the whole stream lifecycle, which a long-lived compiled server does naturally.

Latency-based routing with Thompson Sampling

Requesty's latency policy is not a static leaderboard. Every 60 seconds, an accounting service aggregates the last hour of production traffic from ClickHouse into log-normal TTFT distributions per (provider, model, streaming mode, input-size bucket), with exponential decay weighting and p5/p95 outlier trimming. The router holds these distributions in memory and, per request, draws a Thompson sample from each candidate's posterior: the lowest sampled latency wins. Providers with few observations get wide posteriors, so the router keeps exploring without sacrificing the median request. This is a statistics engine running inside the request path at nanosecond cost, because the distributions are just structs in process memory.

Streaming as a first-class citizen

The chat handler writes through a FlushableResponseWriter that flushes every SSE chunk as it arrives from the provider. Time to first token is the metric our routing optimizes and the one your users feel; a gateway that buffers chunks, or whose event loop hiccups under load, shows up directly in TTFT. Goroutine-per-stream means one slow client back-pressures its own stream and nothing else.

Regional endpoints from the same binary

Because the router is one static Go binary in a Helm chart, standing up a regional satellite is configuration: set region: eu-central-1 and the chart wires up the regional hostnames. That is how eu.router.requesty.ai exists: the same code, deployed in Frankfurt, processing EU traffic entirely inside the EU with zero data retention, routing to EU deployments on Bedrock (eu-central-1), Azure (France Central, Sweden Central), and Vertex. The Frankfurt gateway adds roughly 8ms of routing latency. Try shipping that data-residency guarantee on a runtime you do not control the geography of.

An audit trail for the control plane

Governance is only as good as its paper trail. Requesty's management API writes structured audit logs for every control-plane mutation: API key lifecycle, BYOK provider keys, groups and members, routing policies, prompts, access lists, MCP configuration, auto-topup changes. Each entry records who changed what, with typed per-resource formatters, and the log is queryable per organization. When a security review asks "who widened this model allowlist and when," the answer is a query, not an archaeology project.

Boring, fast operations

The whole router compiles in seconds, ships as a container measured in tens of megabytes, starts in milliseconds, and is profiled with pprof when we want another millisecond back. Go's garbage collector has sub-millisecond pauses at our heap sizes; in three years it has never been the bottleneck. The bottleneck is always the provider, which is exactly how a gateway should fail.

Why Go and not Rust?

Rust gateways are excellent, and the benchmarks above show it. If we were building a CPU-bound inference engine, we would pick Rust. But a gateway is I/O-bound: the compute per request is parsing JSON, evaluating a policy, and shuffling bytes. The measured difference between Go and Rust here is roughly one millisecond, invisible next to a 500ms model TTFT.

What is visible is iteration speed. The provider landscape changes weekly: new models, new API shapes, new failure modes to translate into fallback semantics. Requesty shipped adapters, guardrails, prompt caching, and routing policies at a pace that a borrow-checker tax would have slowed. Go's compile-run-test loop is seconds; onboarding an engineer to productive contribution is days. LiteLLM's Rust rewrite took a team that already had years of gateway domain knowledge; Helicone's took a dedicated effort by infrastructure veterans. Go gets you 95% of the performance for perhaps a third of the engineering cost, and in a category where the moat is routing intelligence and provider coverage rather than raw proxy speed, that trade is correct.

The takeaway

Look at where the category is converging. Every gateway that started in an interpreted runtime and grew serious production traffic has either rewritten its data plane in a compiled language (Helicone: TypeScript to Rust; LiteLLM: Python to Rust, in beta) or is architecturally locked into a platform that trades away warm state and connection pooling for edge proximity that inference workloads cannot use (OpenRouter, Cloudflare AI Gateway). Every gateway born with the benefit of hindsight (Bifrost, TensorZero, LangDB, Envoy AI Gateway, Higress, Requesty) started compiled, and half of them started in Go.

The physics of the workload picked the language. Thousands of concurrent multi-minute token streams, nanosecond-hot shared caches, mid-stream failover, async accounting, single-binary regional deployment: that is a Go program. We wrote it as one, and the 16ms it costs you buys failover across 300+ models, Thompson-sampled latency routing, prompt caching, guardrails, a full audit trail, and an EU region where your data never leaves Frankfurt.

Change your base URL to router.requesty.ai (or eu.router.requesty.ai) and keep your OpenAI SDK. The gateway should be the most boring, fastest thing in your stack. Build it, or buy it, compiled.