LayerNorm vs BatchNorm: why Transformers normalize per token, not per batch

📏 Play with the LayerNorm vs BatchNorm visualizer: https://dev48v.infy.uk/dl/day27-layer-norm.html Normalization inside a neural net is almost embarrassingly simple: take some numbers, subtract their ...