AI Gateways Are Not I/O-Bound Proxies I Benchmarked 5 of Them to Prove It

The author challenges the assumption that AI gateways are simple I/O-bound reverse proxies, demonstrating through benchmarks of five open-source gateways that per-request CPU work (JSON parsing, rate limiting, routing, header mutation, logging) serializes under concurrent load. Using a fixed 60ms mock upstream on identical hardware, the results show widely divergent performance: Ferro Labs and Kong scale linearly, while Bifrost suffers connection pool starvation, LiteLLM hits a CPU ceiling around 175 RPS, and Portkey's latency degrades significantly above 150 VUs.