Quantization is the reason you can run a capable language model on a gaming GPU instead of a rack of accelerators. It stores each weight in fewer bits — 4 instead of 16, typically — which cuts memory roughly fourfold and speeds up inference, because on modern hardware LLM generation is limited by memory bandwidth rather than raw compute.
The confusing part is the format zoo. This explains the memory math, decodes the naming, and tells you which format to use with which stack.
The Memory Math
One formula covers most planning. Memory for weights ≈ parameters × bits ÷ 8:
| Precision | Bits/weight | 7B model | 13B model | 70B model |
|---|---|---|---|---|
| FP16 / BF16 | 16 | ~14 GB | ~26 GB | ~140 GB |
| 8-bit | 8 | ~7 GB | ~13 GB | ~70 GB |
| 4-bit | ~4.5 (with overhead) | ~4 GB | ~7.5 GB | ~40 GB |
Then add the KV cache, which grows with context length and concurrent requests, plus framework overhead. A practical rule: take the weight figure and leave 20–30% headroom. If the result does not fit your card, you need a smaller model or a smaller quantization — not optimism.
Why 4-bit is not exactly 4 — quantized formats store scaling factors alongside the weights, and some layers are kept at higher precision. Real 4-bit files land nearer 4.5–5 bits per weight, which is why a 7B Q4 file is around 4 GB, not 3.5.
Decoding GGUF Names
A file called model-Q4_K_M.gguf is not arbitrary. Reading it left to right:
Q4 — roughly 4 bits per weight — the dominant factor in file size.
_K — a k-quant: precision is allocated per block, with more bits given to weights that matter most. Prefer these over the older non-K types.
_S / _M / _L — small, medium, large variant within that bit width. Larger keeps more layers at higher precision, costing a little size for a little quality.
| Quant | Size | When to choose it |
|---|---|---|
Q8_0 | Largest | Near-lossless; you have VRAM to spare |
Q6_K | Large | Very close to full quality |
Q5_K_M | Medium | Excellent quality/size balance |
Q4_K_M | Small | The default recommendation for most people |
Q3_K_M | Smaller | Only to fit a model that otherwise will not |
Q2_K | Smallest | Noticeable quality loss; last resort |
The Formats and Their Ecosystems
Format choice is mostly determined by which runtime you use — they are not interchangeable:
| Format | Runtime | Notes |
|---|---|---|
| GGUF | llama.cpp, Ollama, whisper.cpp | Only format that splits cleanly across CPU and GPU |
| GPTQ | vLLM, transformers, TGI | GPU-only, post-training quantization; long established |
| AWQ | vLLM, transformers, TGI | Activation-aware; protects weights that matter most |
| bitsandbytes (NF4/INT8) | transformers | Quantizes on load — no separate file needed |
| FP8 | vLLM on recent NVIDIA GPUs | Hardware-accelerated; needs newer datacenter cards |
Practical shortcut — running Ollama or llama.cpp? Use GGUF. Running vLLM or TGI? Use AWQ or GPTQ. That single rule resolves most of the confusion.
bitsandbytes is the convenience option — it quantizes standard weights at load time, so you skip hunting for a pre-quantized file. It is generally slower at inference than a purpose-built quantized format.
Bigger Model or Better Quantization?
The recurring question: with a fixed memory budget, run a 13B model at 4-bit or a 7B model at 8-bit? The widely reported experience in the local-LLM community is that the larger model at lower precision usually wins — down to about 4 bits, below which quality degrades quickly.
Treat that as a starting heuristic rather than a law. Quantization damage is uneven: it affects reasoning and code generation more than casual conversation, and small models suffer proportionally more than large ones. If your workload is precise — generating shell commands, editing configuration — verify quality on your own prompts before trusting a heavily quantized model. See our Ollama guide for the model-sizing side of this.
Quantizing a Model Yourself
Most popular models already have community quantizations published, so this is usually unnecessary. You need it for fine-tuned or private models. With llama.cpp:
# 1. Convert Hugging Face weights to GGUF at 16-bit
$ python3 convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf --outtype f16
# 2. Quantize to the target format
$ ./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
# List every supported type
$ ./build/bin/llama-quantize --help
Keep the f16 file if you have disk space — re-quantizing from it takes minutes, while re-downloading and re-converting the original weights takes far longer.
Verifying You Got What You Expected
# Ollama: inspect a model's quantization and parameters
$ ollama show llama3.2
# llama.cpp prints the quantization type and size on load
$ ./build/bin/llama-cli -m models/model-Q4_K_M.gguf -p "test" -n 1
# Watch actual VRAM use during generation
$ watch -n1 nvidia-smi
The load output tells you the type and how many layers reached the GPU. If VRAM use is far below what you calculated, layers are silently running on CPU — the usual explanation for "why is this so slow".
Common Mistakes
Sizing for weights only — the KV cache is not optional. A model that loads fine can still OOM once the context fills.
Mixing format and runtime — vLLM will not load a GGUF file and llama.cpp will not load an AWQ checkpoint. Match the format to the runtime first.
Going below 4-bit reflexively — Q2 and Q3 exist for cases where nothing else fits. Reach for a smaller model at Q4 before a large model at Q2.
Assuming quality is uniform — quantization hits structured output, code, and long reasoning hardest. Test the task you actually care about.
Ignoring context length cost — raising context from 4K to 32K can add more memory than the difference between two quantization levels.
Trusting a partial download — truncated multi-gigabyte files produce gibberish rather than a clean error. Verify sizes and hashes.
Frequently Asked Questions
What does Q4_K_M mean?
Q4 means roughly 4 bits per weight, _K indicates a k-quant that allocates precision per block rather than uniformly, and _M is the medium variant which keeps more layers at higher precision than _S. It is the most commonly recommended balance of size and quality.
How much VRAM does a quantized model need?
Approximately parameters × bits ÷ 8, plus the KV cache and overhead. A 7B model at 4-bit needs roughly 4 GB for weights, so plan for about 5–6 GB total. Leave 20–30% headroom above the weight figure.
Is a bigger model at 4-bit better than a smaller model at 8-bit?
Usually yes, down to about 4 bits — that is the common experience in the local-LLM community. Below 4-bit quality degrades quickly. Because quantization affects code and reasoning more than casual chat, verify on your own workload before relying on it.
Which quantization format should I use?
It follows your runtime. Use GGUF with llama.cpp or Ollama, and AWQ or GPTQ with vLLM or TGI. bitsandbytes is a convenience option in transformers that quantizes standard weights at load time without a separate file.
Does quantization make inference faster?
Generally yes. LLM token generation is largely limited by memory bandwidth, so reading fewer bytes per weight speeds it up in addition to reducing memory use. The gain depends on hardware and how much of the model fits in VRAM.