Consider an integer, e.g. 0, 1, 2, or 3. It is internally represented by a set number of bits in the computer. With two bits, we can represent the unsigned (positive) integers 0,1,2,3 (00, 01, 10, 11). Given one more bit that determines if the integer is positive or negative, we can now represent signed integers.

For a full precision floating point number, like float32 (4 bytes), we partition the 32 bits into three parts:

sign

with 0 corresponds to positive and 1 to negative number.

exponent

a number of bits (e.g. 8) are reserved for representing the exponent

mantissa

the remaining of the bits (e.g. 23) are used for the significand, meaning representing the significant digits. The precision of the number depends on the number of bits used to represent it.

See this for in depth explanation, and huggingface blogpost on quantization on ML.

Thus we can represent numbers like (using base 10):

• \(61000 \rightarrow 1 \times 6.1 \times 10^{4}\)
• \(-0.0061 \rightarrow -1 \times 6.1 \times 10^{-3}\).

A model like LLaMA2 has 7 billion parameters, thus with 32 bit float, it takes \(\frac{7 \times 10^9 \times 32 }{ 8 \times 10^9}=28\) GB of disk space. If we would express each parameter using fewer bits, it would save resources.

Rather than allocating 64 bits (float64) or 32 bits (float32) full precision for each element in the weight matrix, we can map them to fewer, like float16 (half precision) or int8, with no significant loss of precision.

Note: speed gain e.g. from float32 to int8 depends on GPU. For some there might be no difference, though typically one should aim to get 4 X gain. Ideally, this is combined with a reduced precision execution engine.

There are several ways to map the higher numbers to lower quantization. At the most fundamental level, there are two base cases:

PTQ

Post-Training Quantization, where the weights are already trained, and then cast to lower bit representation.

QAT

Quantization-Aware Training, where quantization is done pre-training, or during fine-tuning, resulting in less performance loss, at the cost of computation. This can be done by inserting quantization & de-quantization between each layer, thus the cost function adapts to it, e.g. by selecting a minima that is broader / less sensitive to the exact value of the weights.

Which to quantize?

If we can reduce a float32 or even a float64 model to int8 model with maintained precision, then the obvious question is:

• How much further can we quantize a floating precision model?

We glean possible answer:

• The paper BitNet: Scaling 1-bit Transformers for Large Language Models (arXiv), they train a model (from scratch) to one bit weights ({-1, 1}), significantly reducing memory and energy consumption, while maintaining a performance comparable to its floating precision counterpart.

• The paper The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (arXiv) explores using a ternary bit ({-1, 0, 1}) weights by adding in the 0 (i.e. log(3)/log(2) bits). They write:

  [It] retains the benefits of the original 1-bit BitNet, including its new computation paradigm, which requires almost no multiplication operations for matrix multiplication and can be highly optimized. Additionally, it has the same energy consumption as the original 1-bit BitNet and is much more efficient in terms of memory consumption, throughput and latency compared to FP16 LLM baselines.

There are pre-trained models on HuggingFace Llama 1 bit and 2 bit models

Different quantization strategies vary greatly in speed and hardware requirements, some taking many hours, others only minutes, depending on if we need to re-train the model or if we can simply operate on the weights. In general, weight quantization comes in two flavors:

Data-free calibration

(e.g. bitsandbytes), using only the weights, and

Calibration-based

(e.g. GPTQ, AWQ), relying on external dataset (risk of introducing bias, and are computationally heavy). AWQ is the latest, and assumes not all weights are of equal importance, to gain inference speed up

Large models like Mixtral-8x7B perform well under 2-bit quantization. However, smaller models like Llama-2-7B do not respond as well to the same treatment, and quantizing down to single bit significantly reduces performance further.

Some quantization methods take hours (depending on zero-shot or one-shot), where e.g. Half-Quadratic Quantization (HQQ), (github) can quantize an LLM in minutes, while BitNet trains the full network from scratch. Furthermore, reducing the Llama-2-7B to 2 bit using HHQ+, yields a model that can outperform the full precision model(!) e.g. on GSM8K (8.5K high quality linguistically diverse grade school math word problems).

Authors behind HQQ+ note:

On one hand, training smaller models from scratch offers the advantage of reduced computational requirements and faster training times. Models such as Qwen1.5 have shown promising results and can be attractive options for certain applications. However, our findings indicate that heavily quantizing larger models using techniques like HQQ+ can yield superior performance while still maintaining a relatively small memory footprint.

Most of the weights of each layer are distributed around the mean. However, when we use a quantization algorithm like absmax in the presence of outliers, they will compress the range for most weights such that most are put in the same bin (of the 8 bins, assuming int8 quantization), such that we can no longer distinguish them.

The outlier problem tends to grow for larger models with more parameters that are to fit into same number of bins. The result is the quantized weights for both outliers and non-outliers will have large error.

This is alleviated by treating outliers separately as float16, as demonstrated in the paper LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv):

• Given a threshold, extract outliers from input matrices
• Multiply non-outliers using int8 vector-wize quantization
• Dequantize int8 to float16 and add to outlier matrix.

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers, (arXiv) 2022

• Initialize with same architecture as original
• Each layer is trained to replicate output from original full precision model
• A fine-grained hardware-friendly quantization scheme for both weight and activations;
• A novel affordable layer-by-layer knowledge distillation algorithm (LKD) even without the access to the original training data;
• A highly-optimized quantization system backend support to remove the quantization/dequantization overhead

Also known as Unstructured pruning, convert all weights below a threshold to zero. This reduces storage requirements.

1. Pick a pruning factor X
2. In each layer, set lowest X % of weights, by absolute value, to 0
3. Optional: retrain model to recover accuracy

Similarly, during training, we can use L1 or L2 regularization to penalize non-zero weights.

Note: In order to actually get an improvement in computational resources, one must use a sparse execution engine, that makes use of the sparsity of the weight matrix.

Storing and computing (unstructured) matrix with zeros takes same amount of time, unless we convert the matrix to some sparse representation, e.g. Sparse MatMul.

Enforce a structured pattern in the network on which weights are allowed to be zeroed out. Thus we remove entire layers or components from the model, that can directly lead to computational speed up.

Consider a N:M structured sparsity pattern:

• For each block of M consecutive values in a matrix (row), only N are allowed to be non-zero
• Only store the values we keep (don’t store zero elements)

Can be done with either:

One-shot pruning

Prunes fully-trained network

Iterative pruning

Prune network gradually

Benefit:

• Good for Nvidia Tensor Core efficiency

In the paper SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (arXiv) they introduced a pruning method tailored for GPT LLM-models but also works with other models.

We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy.

• Can achieve 50% pruning, while magnitude pruning can only reach 10% before performance hit.
• No retraining
• Layer by layer pruning, then stitched together in the end
• Requires expensive weight update process, (inversion of second-order Hessian matrix)

• No weight updates
• No retraining

Wanda: Prune weights based on magnitude and input activation:

• Prune weights by magnitude and input activations

Done by:

• Pruning metric - how to compare the Weights
• Pruning granularity - which weights to compare

• set sparsity level, e.g. 50%, removing half of all weights on each row, determined by size of weight x input activations
• repeat for each hidden layer
• input activations determined by calibration data set
• does not need to update weights after pruning, unlike structured pruning