Every large model release eventually spawns a second wave of smaller, “quantized” versions that somehow run on a laptop or phone with most of the original capability intact. This isn’t a trick or a downgrade in the way it sounds. It’s a deliberate, well-understood technique for trading numerical precision for speed, memory, and cost, and it’s become one of the most important levers in making AI inference affordable.

The core idea: fewer bits per number

Neural networks are, underneath everything, enormous collections of numbers: weights and activations that get multiplied and added together billions of times per forward pass. Those numbers are typically stored as 32-bit or 16-bit floating point values during training, because training needs the precision to make small, accurate updates over millions of steps.

Inference is a different problem. You’re not updating anything, you’re just running the math forward. Quantization asks: do we really need 16 or 32 bits of precision to represent each weight, or can we get away with 8 bits, or 4, and still get answers that are close enough to correct?

In practice, the answer is often yes. A weight matrix quantized to 8-bit integers takes a quarter of the memory of its 32-bit float counterpart, and integer arithmetic is generally cheaper and faster on most hardware than floating point. Cut to 4 bits and you’re at one-eighth the memory footprint, though the accuracy tradeoffs get more noticeable.

Why this doesn’t just break the model

The naive intuition is that chopping precision should degrade output quality roughly in proportion to how much precision you removed. That’s not really how it plays out, for a few reasons.

Neural network weights tend to be redundant. Many models are meaningfully overparameterized relative to the task, so small perturbations in individual weight values average out across a layer’s worth of multiply-accumulate operations rather than compounding into visible errors. A model can tolerate a fair amount of per-weight noise before the aggregate output shifts enough to matter.

Quantization schemes also aren’t naive rounding. Most practical approaches calibrate scale factors per layer, or even per channel, based on the actual distribution of values in that layer, so the available integer range is spent where the weights actually live instead of being wasted on values that never occur. Some methods go further and quantize weights and activations differently, or keep a small number of sensitive layers at higher precision while aggressively quantizing the rest. This is why you’ll see labels like “8-bit weights, 16-bit activations” rather than a single blanket precision level.

There’s also a difference between quantizing after training (post-training quantization) and quantizing as part of training or fine-tuning (quantization-aware training). The latter lets the model adjust its weights to be more tolerant of the eventual precision loss, which generally preserves more quality at very low bit widths, at the cost of extra training work.

What you actually get for it

The payoff isn’t just a smaller file on disk. Lower-precision arithmetic runs faster on hardware that supports it, memory bandwidth requirements drop, and more of the model can fit in fast cache or on-device memory instead of spilling to slower storage. For inference serving at scale, that translates directly into lower latency and lower cost per request, since memory bandwidth and capacity are frequently the actual bottleneck rather than raw compute. For on-device use cases, it can be the difference between a model fitting on a phone or an embedded chip at all versus needing a data center GPU.

Where the tradeoffs show up

Quantization isn’t free, and the losses aren’t always uniform across tasks. A model quantized aggressively might perform nearly identically on common, well-represented queries while degrading more noticeably on edge cases, rare vocabulary, or tasks requiring fine-grained numerical reasoning, since those are exactly the situations where small precision errors are more likely to flip an output. This is part of why teams doing this work run task-specific evaluations rather than relying on a single aggregate quality score, and why a quantized model that looks fine in a quick benchmark can still surprise you on a narrower workload it wasn’t tested against.

There’s also a practical ceiling. Below a certain bit width, generally somewhere around 4 bits for many current architectures, quality tends to degrade faster than the memory savings justify, though this threshold shifts as calibration and training techniques improve. Choosing a quantization level is ultimately a judgment call about which point on that curve matches the deployment target, whether that’s a cloud inference cluster optimizing for throughput or a mobile app optimizing for whether the model runs at all.

Quantization won’t replace the need for genuinely smaller, more efficient model architectures, but as a way to get more out of models that already exist, it’s one of the more reliable tools available, and it’s a big part of why “runs locally” has become a realistic claim rather than an aspiration.