When GPU Acceleration Backfires: The Dispatch Overhead Problem

GPU acceleration is conventionally recommended for all machine learning inference, but benchmarks reveal a critical oversight: for small embedding models processing single queries, GPU dispatch overhead exceeds performance gains. This finding challenges universal optimization advice and demonstrates that hardware decisions must align with workload characteristics—batch size, model size, and sequence length. Understanding when dispatch costs outweigh computational benefits is crucial for production retrieval systems seeking genuine latency improvements.