1

Introduction to Large Language Models System

Zirui “Ray” Liu

University of Minnesota, Twin Cities

Quantization

2

Outline

• What is quantization? Why we need it
• Implement Quantization on Hardware Accelerator
• aka quantization kernel
• What is the “tax” you pay for quantization?
• Quantization Algorithm

3

Takeaway for quantization

• For weight-only quantization, you should use asymmetric quant
• Cannot use INT tensor cores
• Reduce latency in small batch setting
• But slow down your inference in larger batch setting
• Easier to quantize compared to activation-weight co-quant
• For activation-weight co-quant, you should use symmetric quant
• Can use INT tensor cores
• Reduce latency and improve throughput

4

Quantization

• Lossy compression technique to reduce the numerical precision
• Why you need it (compact your model):
• 1B param in FP16 2 GB
• 1B param in FP8 1 GB
• 1B param in FP4 0.5 GB

5

Quantization

• Lossy compression technique to reduce the numerical precision
• Why you need it (hardware friendly):
• Much higher throughput when you use low bit arithmetic

6

Quantization

• Lossy compression technique to reduce the numerical precision
• Why you need it (much lower energy consumption):

7

IEEE 754 Floating Point Number

8

Integer Number

9

Asymmetric v.s. Symmetric Quantization

10

Asymmetric v.s. Symmetric Quantization

• Asymmetric quant. is more “accurate” than symmetric quant.
• Symmetric quant. can be covered by Asymmetric quant

11

Weight only quantization kernel

• Use case: Single request, edge devices, small batch
• We should use Asym. quant for weight only quantization
• (sym. quant is also okay, but no benefits)
• Explicitly dequant & FP16 Matmul
• Fuse the dequant & matmul

12

Weight only quantization kernel

• Extra FLOPs due to dequantization
• I/O between HBM and SRAM is saved
• Suitable for memory-bounded OPs

13

Weight only quantization kernel

• Benefits smaller batch inference (memory bounded)
• Slow down server side inference (extra FLOPs)

14

Activation-Weight quantization kernel

• Use case: Batch request, server side, chatbot service
• Both X and W must be quantized symmetrically
• Can use INT tensor cores

15

Activation-Weight quantization kernel

• How to quantize X & W, specifically:
• You should quantize X & W along the outer dim, symmetrically
• If you do not do it in this way, quant. will slow down inference

​

​

16

Activation-Weight quantization kernel

• Use case: Batch request, server side, chatbot service
• Reduce I/O between HBM and SRAM
• Can use INT Tensor Cores
• Reduce both latency and

improve throughput

• TL; DR: larger # data trained, higher numerical precision
• If learning is compression (token ID can be represented in uint32, 4bytes):
• LLama-1: 1T tokens -> 7B weights

(3725 GB of tokens -> 14GB weight file) 266 X

• LLama-2: 2T tokens -> 7B weights

(7450 GB of tokens -> 14GB weight file) 532 X

• Llama-3: 15T tokens -> 8B weights

(55875 GB of tokens -> 16GB weight file) 3496 X

Insight regarding “Compression”

• TL; DR: larger # data trained, higher numerical precision
• If learning is compression (token ID can be represented in uint32, 4bytes):
• LLama-1: 1T tokens -> 7B weights

(3725 GB of tokens -> 14GB weight file) 266 X

• LLama-2: 2T tokens -> 7B weights

(7450 GB of tokens -> 14GB weight file) 532 X

• Llama-3: 15T tokens -> 8B weights

(55875 GB of tokens -> 16GB weight file) 3496 X

Compression is hard to do for latest models

19

Takeaway for quantization

• For weight-only quantization,
• Use case: edge devices, small batch
• Asym. quant
• For activation-weight co-quant,
• Use case: server side, large batch
• symmetric quant

20

Quantization Algorithm

• Change the “Round to Nearest” Strategy to an optimization based one

21

Quantization Algorithm

• Change the “Round to Nearest” Strategy to an optimization based one