Teaching Models to Continuously Grow Without Losing What They Know

Self-Distillation Fine-Tuning (SDFT) addresses a critical bottleneck in foundation models: acquiring new skills while preserving existing capabilities. Unlike traditional supervised fine-tuning, SDFT leverages in-context learning to generate on-policy training signals, enabling models to accumulate multiple skills over time without catastrophic forgetting.