Quantization Formats Guide: GGUF vs GPTQ vs AWQ — Which File?

You open Hugging Face to run a local LLM, and the same model has a wall of files—Q4_K_M, Q5_K_S, GPTQ, AWQ, IQ3_M… and you freeze. It's the first wall most people hit. This article answers, practically, "which quantized file do I download to make it run?" We leave what quantization is (the concept) to another article and focus here on choosing the format.

We'll cover why the format is set by the runtime, the GGUF naming scheme (Q4_K_M), the 4 formats compared, how to pick the bit depth, and how to find files. The key points up front. ① The format is mostly decided by "which engine you'll run it on" (GGUF = llama.cpp/Ollama; GPTQ/AWQ = GPU engines like vLLM). ② Q4_K_M means "4-bit, K-quant, size M"—the common sweet spot in most cases. ③ When in doubt, start with Q4_K_M; step up to Q5/Q6 if you have the VRAM.

1. Why quantized files are confusing

Quantization means rounding a model's weights to fewer bits to cut size and memory (see what quantization is). The catch is that there are several methods (formats) for that rounding, and each has many variations in bit depth and granularity. The result: 20-plus files under one model on Hugging Face.

But relax—the choice becomes easy once you split it into two steps: ① which format (= which engine you'll run it on), then ② which bit-depth file within it. Think in that order.

2. The key rule: the format is set by the runtime

The most important fact: a quantized file only runs on engines that support its format. So decide the engine first, and the format is almost automatic.

Remember one thing: GGUF is the only "local do-it-all" format that runs on CPU / Mac / partial-GPU; the rest (GPTQ/AWQ/EXL2) are basically GPU-only. So—

• Running on your own PC / Mac / CPU (Ollama or LM Studio) → GGUF, no contest.
• High throughput on a GPU server (vLLM/TGI serving many requests) → GPTQ or AWQ.
• Squeezing bit width finely on a single GPU → EXL2/EXL3.
• Quick test in Transformers (no pre-quantized file) → bitsandbytes (NF4).

3. Decoding GGUF names (Q4_K_M)

GGUF, which you'll meet most for local use, is harmless once you can read the "cipher" in the filename. Q4_K_M is made of three parts.

Two more terms keep you from getting lost.

• The IQ family (IQ2_XS, IQ3_M, etc.) = I-quants: a newer, codebook-based line that can go even smaller at the same bit depth. But inference is heavier, and they practically require the imatrix below. A trump card for "I really need to fit this in VRAM."
• imatrix (importance matrix): run a calibration text through the model to measure which weights matter, then protect those first at low bit depth. Optional for K-quants, effectively required for I-quants. GGUFs built with imatrix hold up better at low bits.

4. The 4 formats compared (GGUF/GPTQ/AWQ/EXL2)

Here's the character of each major format, GPU ones included.

5. Which file / bit depth to pick

Once the format is set, next is the bit depth. Using GGUF as the example, here's a practical ladder for "when in doubt, pick this" (the numbers are approximate and vary by model and build).

As a rule of thumb, people often say about 4.5–5 bits per weight (Q4_K–Q5_K) is the "tasty" band (a heuristic). Above Q6 the quality gains slow down, and Q8_0 or FP16 cost a lot in memory and speed for little difference—that's the practical consensus. Start by running Q4_K_M, step up to Q5/Q6 if it feels short, drop to IQ if it won't fit, and tune as you go. For estimating the VRAM you need, see local LLM hardware requirements.

6. Finding files on Hugging Face and Ollama

On Hugging Face, filter GGUF with library=gguf, or go to a quant re-packager's repo. As of writing, bartowski and mradermacher actively publish GGUFs (including imatrix builds)—but re-packager activity changes, so check the latest upload date. (TheBloke's once-standard repos still exist, but new uploads have largely stopped.) To convert your own, the official gguf-my-repo Space works.

On Ollama, tags follow model:size-variant-quant.

# No tag = the default (Q4_K_M for many models)
ollama pull llama3.1

# Choose the quant explicitly
ollama pull llama3.1:8b-instruct-q5_K_M
ollama pull qwen2.5-coder:7b-instruct-q8_0

# See available tags at ollama.com/library/<model>/tags

So the real workflow is three moves: decide the engine → narrow to its format → pick the bit-depth file that fits your VRAM. Master that and the 20-file wall stops fazing you. For actually running one, see how to run a local LLM and the Ollama guide.

Summary

Choosing a quantization format is two steps. First ① which engine you'll run it on—local CPU/Mac → GGUF; GPU server for speed → GPTQ/AWQ; fine-tuned bit width on one GPU → EXL2; quick Transformers test → bitsandbytes. The top rule: a format only runs on engines that support it.

Then ② the bit depth. GGUF's Q4_K_M means "4-bit, K-quant, size M"—the common default. When in doubt, Q4_K_M → (if you have room) Q5/Q6 → (if it won't fit) IQ. Anchor on the ~4.5–5 bpw "tasty band" heuristic and tune while running. imatrix builds hold up better at low bits. Related: what quantization is, how to run a local LLM, hardware requirements, best local models, Ollama guide.

FAQ

Q. So which file should I pick?
A. First decide "which engine you'll run it on." Your own PC or Mac (Ollama/LM Studio) → GGUF; a GPU server (vLLM/TGI) → GPTQ or AWQ. Then for bit depth, when in doubt, Q4_K_M (good size/quality balance, and Ollama's default for many models). With VRAM to spare, Q5_K_M/Q6_K; only when you must fit a big model, consider IQ2/IQ3.

Q. What does Q4_K_M mean?
A. Three parts. Q4 = ~4-bit (higher number = higher quality, bigger file), K = K-quant (smart quantization over super-blocks; plain or _0/_1 are legacy), and M = size medium (S/M/L is how much some important tensors get upgraded to higher bits). Effective bits run a touch above the label (Q4_K ≈ 4.5 bpw).

Q. How do GGUF and GPTQ/AWQ differ?
A. By the runtime (hardware) they support. GGUF is the only "local do-it-all" format that runs on CPU, Mac, and partial-GPU, for llama.cpp/Ollama. GPTQ and AWQ are GPU-first and suit high throughput on vLLM/TGI. They also differ in method (GPTQ minimizes layer-wise error; AWQ protects salient weights via activations), but neither is universally better. Just pick whichever your engine supports.

Q. How is IQ (I-quant) different from a normal Q4?
A. It can go even smaller at the same bit depth (a newer, codebook-based method). In return, inference is a bit heavier, and it practically requires an imatrix to hold quality. Its use is a trump card for "I must fit a big model into VRAM." If it fits normally, a K-quant like Q4_K_M is easier to work with.

Q. Isn't Q8_0 or FP16 higher quality? Why "not recommended"?
A. They are indeed near-lossless, but they eat far more memory and run slower for only a tiny quality gain over Q5/Q6. Even the llama.cpp community doesn't recommend them for normal use. In practice, about 4.5–5 bpw (Q4_K–Q5_K) is the "tasty" band, and you move up or down from there as needed (numbers are approximate and vary by model/build).