What is Quantization?

Quantization shrinks an AI model by storing its numbers with fewer bits. The model is the same design with the same learned knowledge, but each number is rounded to a coarser value, so the file is smaller and easier to run.

How it works

A trained model is a very large collection of parameters, and the file that holds them is the model weights. Each weight is a decimal number. Models are commonly trained and shipped at 16-bit precision. Quantization converts those numbers to 8-bit, 4-bit or even fewer bits.

Think of a photo saved as a smaller JPEG. You lose some fine detail, but the picture is still recognizable and the file is far smaller. Fewer bits per weight means a smaller file and less memory used while the model runs.

A rough rule: halving the bits roughly halves the memory the weights need. A model that needs 16 GB at 16-bit needs about 8 GB at 8-bit and about 4 GB at 4-bit, plus extra memory for the conversation itself.

Why it matters to you

Memory is usually the limit on running a model yourself. Quantization is why large open-weight models can run on a laptop, a gaming graphics card or a desktop instead of a data-center server. It also makes inference cheaper and often faster, because there is less data to move around for every token the model writes.

The trade-off

Rounding costs accuracy. At 8-bit the loss is usually small. At 4-bit it is often acceptable for chat and coding. At 2 or 3 bits, quality can drop noticeably, with more mistakes and weaker reasoning. How much you lose depends on the model and the method, so test a quantized build on your own tasks before relying on it.

Quantization methods differ. Some round every weight the same way, while others protect the weights that matter most. You will see names like GGUF, GPTQ, AWQ and FP8 on download pages. They are different formats and methods for the same basic idea.

Quantization versus related ideas

• Distillation trains a new, smaller model to imitate a bigger one. Quantization keeps the same model and only lowers its precision.
• Fine-tuning changes what the model has learned. Quantization changes how precisely it is stored.
• Quantization can also be applied after training, which is called post-training quantization, or planned for during training.

A quick example

Say you download a model and your graphics card has 8 GB of memory. The full-precision version will not fit. A 4-bit quantized version of a smaller model might, and it will run, with a small drop in quality. That is the whole bargain: a little accuracy for a lot of reach.