The Mean Is Lying to You: Benchmarks Hide the Variance That Breaks Prod

TL;DR— Benchmark scores report central tendency over a fixed, static distribution of test items, but production reliability is governed by tail behavior on a shifting distribution of real inputs. A mo...