The Three Layers of LLM Infrastructure: Gateway, Proxy, and Router

Industry products like LiteLLM and Portkey use 'gateway' and 'proxy' interchangeably, creating confusion about their actual roles. This article clarifies that these are three distinct layers of the same stack—Proxy (transport layer), Router (decision layer for intelligent model selection), and Gateway (policy layer for auth and rate limits)—with practical Go examples for each. Understanding this architecture is critical for teams optimizing LLM cost efficiency, reliability, and operational compliance.