Speculative Decoding Brings 3x Speed Boost to Mobile LLM Inference

Speculative decoding can reduce LLM generation latency by 2-3x, but implementing it on mobile requires overcoming significant hardware constraints. This workshop demonstrates how to split inference across Android's heterogeneous accelerators—using Vulkan compute for a lightweight draft model and NNAPI for a larger verification model—to achieve sub-200ms per-token generation on flagship devices. The result is genuinely practical on-device LLM inference, opening new possibilities for privacy-preserving AI applications that don't sacrifice speed.