CODA Cuts Memory Overhead in Transformer Training Through GPU Kernel Fusion

CODA addresses a critical performance bottleneck in Transformer training: memory-bound operations that waste throughput despite highly optimized GEMM kernels. By fusing normalization, activation, and residual operations into GEMM epilogue programs, CODA keeps intermediate tensors on-chip longer, dramatically reducing global memory traffic. This work demonstrates how specialized GPU abstractions can bridge the productivity-efficiency gap in deep learning frameworks.