In the Media

Researchers at MIT have “developed a new technique that enables large language models to learn new skills and knowledge without forgetting their past capabilities,” reports Ben Dickson for Venture Beat. “Their technique, called self-distillation fine-tuning (SDFT), allows models to learn directly from demonstrations and their own experiments by leveraging the inherent in-context learning abilities of modern LLMs,” explains Dickson. “Experiments show that SDFT consistently outperforms traditional supervised fine-tuning (SFT) while addressing the limitations of reinforcement learning algorithms.”