One Mathematical Tweak Unmasks Transformers' Hidden Norm Bias

RBF attention proves to be just dot-product attention with an explicit L2 norm penalty, revealing how modern Transformers inherently exploit large-norm keys. This insight explains long-standing mysteries like attention sinks and suggests that fixing attention geometry requires no architectural revolution—only understanding and modifying the scoring mechanism, avoiding costly changes to core systems like FlashAttention.