The Speculative Decoding Trap: How Cached Patterns Fool LLM Benchmarks

A real-world test on consumer GPUs reveals that speculative decoding—a technique claimed to deliver 2-3x inference speedups—largely fails with diverse workloads. The critical discovery: apparent performance gains stem not from genuine speed improvements but from n-gram cache memorization of repeated patterns, a benchmarking pitfall many practitioners overlook. This finding underscores the importance of testing optimization techniques beyond synthetic benchmarks to avoid misleading conclusions about their real-world impact.