Beyond the UNet: Unmasking the Real Bottlenecks in Diffusion Inference

While most developers obsess over UNet attention, real-world inference delays are often hidden in neglected components like the VAE decoder and text encoders. By profiling and optimizing these secondary stages—using techniques like bf16 casting and Tiny AutoEncoders—developers can achieve significant performance gains without touching the core model architecture.