RoPE: How 2D Rotations Solved Transformer Long-Context

Why 2D Rotations Solved Transformer Long-Context: A Mechanics-First Look at RoPE Attention requires two things from positional encodings: - Relative Distance Sensitivity: Token attending to Token shou...