How DistillationTurns AI Models into Teachers by Dr. Timothy Smith

Photo Source: StockVault

The news today contains thousands of stories related to artificial intelligence (AI), and the content varies from the breathlessly excited and encouraging articles on how AI will transform the world to the more negative and apocalyptic articles on how AI will destroy the world as data centers gobble energy and tranquility. Amid the expansion of AI and the furious arms race to build larger, more capable models, clever researchers have found ways to shortcut the costly process of building new AI models by building cheaper ones on the backs of leading models through a process called "distillation." Undoubtedly, the costs continue to climb astronomically. For example, only a few years ago, in 2022, an industry-leading AI model cost anywhere from $ 2-12 million to train. Only four years later, Dario Amodei, CEO of Anthropic, said it cost around $100 million to train a lower-end model and over $1 billion for a frontier model. (entrepreneur.com) According to The Wall Street Journal, Anthropic and OpenAI will spend a combined $64.8 billion on training models this year and expect to spend nearly $250 billion in 2029. (wsj.com)

With the stratospheric costs of new models growing unchecked year on year, model builders with less budget have alternative tools at hand. They can use the knowledge built into leading models to "teach" less advanced, smaller "student" models at a fraction of the cost of building new ones, using a process called distillation. Distillation was first introduced in 2015 in a paper by Google researchers Hinton, Vinyals, and Dean, which describes a system that trains a student model using the complex output of a large teacher model. Instead of learning word by word like a large language model, the student learns the teacher's probability distribution over likely words, saving time and considerable money. (arxiv.org)The teacher model learns by guessing the correct missing word in a sentence. For example, the sentence "The dog ___ the bone" might have the word "chewed" 50% of the time, but other possibilities include "buried, 25%," "licked, 15%," and "hid, 5%." These other possible words make for a distribution of probable words. This distribution comes from the initial training of the teacher model, and the "student" model can take the whole distribution, saving time and reducing computing costs. The student does not have to spend as much time as the teacher to achieve similar capabilities.

As smaller student models learn from teacher models, they can mimic the teacher's responses and imitate the teacher's preferences and flavors. Many student models retain elements and flavors of the teacher, including phrasing, formatting, and even preferences for certain answers to questions. In fact, research shows that student models will retain teacher preferences even if the student model does not receive any specific information suggesting a preference. In an article titled "Subliminal Learning: Language models transmit behavioral traits via hidden signals in data," the authors found that student models will learn preferences such as "likes owls" or display toxic misalignment even if trained only on sequences of numbers unrelated to owls or bad behavior. Misalignment can include suggestions of harmful advice, such as illegal or dangerous activities. The authors claim that AI neural networks experience subliminal learning that training cannot remove. Such deep traits or flavors may propagate from teacher to student in ways we cannot yet predict or remove.

As the AI race to build more powerful models continues, data center and computing costs have driven the price of new models far out of reach for all but a few tech giants and government institutions. Restricting the opportunity to build better models to only a few major players will drive smaller players to seek cheaper, alternative techniques to build models for their uses. Distilling has proven to offer a much cheaper way to take the advances of frontier models and build student models at a fraction of the cost. Although this system works, it can propagate unintended flavors and behaviors from the teacher model to students, preserving and even amplifying benign preferences as well as more dangerous misalignments. Because these flavors transfer subliminally through seemingly unrelated information, the risk of multiplying bad behavior and preferences across many student models remains a growing danger, with more models exhibiting bad behavior landing in the hands of a growing range of users.

Dr. Smith’s career in scientific and information research spans the areas of bioinformatics, artificial intelligence, toxicology, and chemistry. He has published a number of peer-reviewed scientific papers. He has worked over the past seventeen years developing advanced analytics, machine learning, and knowledge management tools to enable research and support high-level decision making. Tim completed his Ph.D. in Toxicology at Cornell University and a Bachelor of Science in chemistry from the University of Washington.

You can buy his book on Amazon in paperback and in kindle format here.