Smaller Models Beat Larger Teachers: TIPSv2's Distillation Breakthrough

TIPSv2 reveals a counterintuitive finding: knowledge distillation produces superior patch-text alignment compared to standard pretraining, allowing smaller student models to outperform their larger teachers. By combining iBOT++, Head-only EMA, and Multi-Granularity Captions, TIPSv2 achieves competitive performance across 9 tasks and 20 datasets while significantly reducing computational costs. This efficiency breakthrough opens new possibilities for deploying vision-language models at scale, with particularly strong zero-shot segmentation capabilities.