FlexLink: Boost GPU Bandwidth by 27% and Accelerate LLM Training by Unlocking Hidden Hardware Pathways

FlexLink addresses the communication bottleneck in distributed LLM training, where inter-GPU data transfer consumes 60-80% of training time despite fast computation. Current libraries like NCCL only use the fastest interconnect (NVLink) while ignoring other available pathways like PCIe, leaving hardware underutilized. FlexLink unlocks these hidden hardware pathways to boost GPU bandwidth by 27% by coordinating multiple heterogeneous communication links simultaneously.