GPU Problem #1: Why Your PyTorch Training Runs Out of GPU Memory (and How to Actually Debug It)

PyTorch training often crashes with CUDA out-of-memory errors despite nvidia-smi showing free memory, caused by memory fragmentation where free memory is scattered in small non-contiguous blocks. Standard tools like nvidia-smi and torch.cuda.memory_summary() only provide snapshots without revealing temporal patterns or root causes. Ingero uses eBPF uprobes to trace every cudaMalloc/cudaFree call at the kernel level with under 2% overhead, exposing allocation imbalances, fragmentation pressure, and the exact Python code paths responsible.