Quantisation

Quantisation is the process of reducing the precision of the numbers used to represent a neural network's parameters. A standard model stores each parameter as a 32-bit floating-point number, which provides high precision but requires significant memory and compute. Quantisation converts these to lower-precision formats — 16-bit, 8-bit, or even 4-bit — reducing the model's memory footprint proportionally. The key insight behind quantisation is that neural networks are remarkably tolerant of reduced precision. A model that is four times smaller due to 8-bit quantisation typically retains 95-99% of its original quality. This makes quantisation one of the most practical techniques for making large AI models accessible on consumer hardware.

There are several approaches to quantisation. Post-training quantisation (PTQ) takes a pre-trained model and converts its weights to lower precision after training is complete. This is the simplest approach and requires no retraining. Quantisation-aware training (QAT) incorporates reduced precision during the training process itself, often producing better results but requiring more effort. Common quantisation formats include GPTQ (GPU-optimised quantisation), GGUF (used by llama.cpp for CPU and mixed inference), AWQ (activation-aware weight quantisation), and bitsandbytes (integrated with popular training frameworks). Each format has different trade-offs in speed, quality, and hardware compatibility. The quality impact varies by model and quantisation level. 8-bit quantisation is nearly lossless for most models. 4-bit quantisation introduces more noticeable degradation but is often acceptable for many practical applications. Below 4-bit, quality drops more sharply.

Quantisation is the enabling technology for running powerful AI models on affordable hardware. A 70-billion-parameter model that normally requires multiple expensive GPUs can run on a single GPU when quantised to 4-bit precision. This dramatically changes the economics of local AI deployment. For organisations concerned about data privacy, quantisation makes it feasible to run capable models entirely on-premises without sending data to cloud APIs. For cost-conscious deployments, it reduces the GPU memory required, allowing smaller and cheaper hardware to serve the same models. Quantisation also improves inference speed — smaller numbers require fewer computation cycles, and more of the model can fit in fast GPU cache memory. This results in faster response times for end users.

Quantisation is widely used in local AI deployments, where organisations run open-source models like Llama or Mistral on their own hardware. Tools like llama.cpp, Ollama, and vLLM support quantised models natively, making deployment straightforward. Edge AI applications — running models on devices like phones, IoT devices, or embedded systems — rely heavily on quantisation to fit useful models into limited memory. In cloud deployments, quantisation reduces the number of GPUs required, lowering infrastructure costs. Many production systems use quantised models for initial responses and fall back to full-precision models only when higher accuracy is needed.