MegaTrain: Breaking the Memory Wall for Single-GPU 100B+ LLM Training

This research shifts the LLM training paradigm from GPU-centric to memory-centric, enabling massive 100B+ models to be trained on a single GPU by treating it as a transient compute engine. By effectively utilizing host CPU memory and optimized pipelining, it significantly lowers the hardware barrier for developing state-of-the-art AI models without needing massive clusters.