What Is Model Compression? Distillation vs. Quantization, Explained Simply

AI models tend to get smarter as they get bigger. But giant models require powerful servers and large amounts of electricity to run, and they’re slow to respond. That’s why compression (model lightening)—making a model smaller and faster while preserving as much performance as possible—matters.

The two representative compression techniques are knowledge distillation and quantization. The names sound difficult, but the ideas are simple once you think of them as “passing knowledge from a smart teacher to a small student” and “moderately reducing the precision of numbers.” This article explains why compression is needed, what knowledge distillation and quantization each do, and how to use them, using original papers and expert explanations and aimed at non-specialists. Reading it alongside our guide to large language models (LLMs) will deepen your understanding.

Why Is Compression Needed?

AX Inc.’s explainer defines a lightweight LLM as “an AI model that reduces model size and computational cost while maintaining performance as much as possible” ².

The same explainer cites three reasons compression is in demand ². First is cost reduction: large models need high-performance GPU servers and large amounts of power, but compression can sharply cut infrastructure costs. Second is response speed: the lighter the model, the faster it reacts, improving real-time performance. Third is expanding where it can be deployed: it becomes easier to run on edge devices (the devices in your hand)—such as smartphones and factory machinery ².

This “able to run locally” point carries real weight when you consider tasks involving confidential data that can’t be sent to the cloud, or use in the field where connectivity is unreliable. Compression isn’t merely about saving money; it’s also a technique for expanding where AI can be used.

Knowledge Distillation: From a Smart Teacher to a Small Student

The origin of distillation traces back to the 2015 paper “Distilling the Knowledge in a Neural Network” by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. The paper noted that making predictions with an ensemble (a collection) of models is “cumbersome and may be too computationally expensive to allow deployment” ¹, and proposed a method to compress that knowledge into a single model that is “much easier to deploy” ¹.

The mechanism is to “transfer the knowledge of a large model (the teacher model) to a smaller model (the student model)” ³. AX Inc.’s explainer similarly describes it as a method to “pass on the knowledge held by a high-performance, giant teacher model to a smaller student model” ². The student model learns by imitating the teacher model’s outputs (its way of answering and tendencies in judgment), thereby acquiring abilities close to the teacher’s in a smaller body.

According to ACTIONBRIDGE’s explainer, the merits of knowledge distillation are “reduced model size and improved inference speed,” while the drawback is that “the distillation process requires additional training time” ³. In other words, you get a light, fast model, but you need a training step to raise the student.

Quantization: Lowering the “Fineness” of Numbers

The other representative technique is quantization. Rather than rebuilding the model’s contents, it shrinks the size by changing how the model’s vast numbers (parameters) are represented.

AX Inc.’s explainer describes quantization as “the process of converting 32-bit numbers into integers with fewer bits” ². For example, converting from 32-bit (FP32) to an 8-bit integer (INT8) shrinks the size to roughly a quarter, and 4-bit (INT4) to roughly an eighth ². ACTIONBRIDGE also calls quantization “a technique for representing model parameters at low precision (e.g., 16-bit, 8-bit),” noting that reducing numeric precision saves memory ³.

The image is similar to slightly lowering a photo’s resolution to make its file smaller. The merits are “reduced memory consumption and improved computation speed,” but in exchange there’s a drawback that “a drop in accuracy can sometimes be observed” ³. It’s a trade-off: the coarser you make the numbers, the lighter it gets, but subtle nuances may be lost.

Differences and How to Use Them

Knowledge distillation and quantization share the same goal of making things lighter, but their approaches differ. Distillation goes in the direction of “having a small student learn from a smart teacher and rebuilding the model,” while quantization goes in the direction of “lowering the fineness of the numbers in an existing model to make it lighter.” There’s also the difference that distillation takes the extra effort of separate training, whereas quantization can be applied to an existing model relatively quickly.

Note that these two are not the only compression techniques. There is also pruning, which slims a model down by “removing low-importance connections within a neural network” ². ACTIONBRIDGE notes that combining these techniques can maximize the effect ³. In practice, approaches like making a small model with distillation and then finishing it with quantization are also used.

Compression is the technology behind the spread of small open models published on Hugging Face and lightweight models like Google’s Gemma. At a time when not just the largest models but also “small, smart models” are drawing attention, keeping the two keywords distillation and quantization in mind gives you a handle on judging which model fits your use case or device. It’s also worth distinguishing that compression is a technique for shrinking an existing model, whereas fine-tuning—which raises performance for a specific task—has a different purpose.