The Scaling Trap: When Distributed GPU Training Backfires

Distributed Data Parallel (DDP) can paradoxically slow training when communication overhead exceeds computational gains, as demonstrated with a 4M-parameter model on two T4 GPUs. The architecture becomes efficient only when model size is large enough that compute time dominates synchronization costs. Understanding this scaling threshold is crucial for practitioners choosing between single-GPU, DDP, and sharded approaches like FSDP.