Quantization reduces the precision of model weights. In local AI, this can make a model small enough to fit on consumer hardware.
In practice
A heavily quantized model uses less RAM or VRAM and may run faster, but it can lose some quality. Common local model files offer several quantization levels so users can choose a tradeoff.
What to watch
The smallest file is not always the best experience. If answers become worse, try a less compressed version or a smaller model at higher precision.