Breaking Down Transformer: How Memory Caching and CTM Challenge Self-Attention

Two groundbreaking research papers—Google's Memory Caching and Sakana AI's Continuous Thought Machine—challenge a fundamental design flaw in Transformers: the coupling of memory and computation in self-attention's O(L²) mechanism. By decomposing this unified design, they offer a more efficient path to handling longer contexts without incurring full attention costs. While these approaches remain in the research phase, they represent a critical shift toward solving one of deep learning's most pressing scalability challenges.