Say you want to run a 30B LLM on your own computer. If your graphics card is a top-tier RTX 5080, 16GB of VRAM is relatively generous. But what do you do when the 30B model file alone is over 20GB and exceeds your VRAM?
You could buy a more expensive graphics card, but there's also a way to store the model with smaller numbers. That's quantization. It expresses the countless numbers a model has learned with fewer bits to cut file size and memory use.
In the past, quantization technology wasn't widespread, so models far larger than your VRAM were hard to even try running, but thanks to quantization, overcoming that problem has become possible.
✦ Key Update
Ollama v0.40.6
Ollama v0.40.6, released on October 16, made an update related to this. The newly supported format is Q2_0. It's one of the GGUF formats that compress model weights into very small units.
Of course, llama.cpp could already run Q2_0 GGUF, but Ollama's file reader couldn't read that format properly. With this update, Ollama can also load and run models in that format cleanly.
How Does Quantization Make a Model Smaller?
A language model is made of billions of weights. You can think of a weight as a number the model uses to understand sentences and pick the next word.
Would it be easier to understand if I called it "a reference number for finding which word is statistically better to pick among countless words"? If these numbers were originally stored as 32-bit floating point, that is FP32, each number needs 4 bytes.
One number looks small, but with billions of them, a single file grows to tens of gigabytes.
✦ What Happens in Quantization
When you shrink the numbers
Quantization reduces the amount of these numbers in a reasonable way. Think of drawing a blueprint with a ruler with fine markings and then switching to a ruler with coarse markings.
The coarser the markings, the easier the work gets, but it becomes harder to draw a very precise blueprint perfectly. There will surely be small gaps.
So quantization always involves a trade-off. In exchange for shrinking files and memory, you give up a bit of the original model's answer quality. In return, since there's less data to read from memory, inference should get faster.
However, it doesn't always get faster. If the device running it (mainly the graphics card) can't process the format efficiently, it takes time to unpack the compressed numbers for computation, and the speed advantage can shrink.
In a war, if you have a codebreaker on your side, you can pass messages quickly in code and turn the tide in your favor, but without a codebreaker, passing coded messages for no reason will have you running out of time decoding them.
In such a case, sending messages in plain text might actually be more efficient. This is also why NVIDIA's RTX graphics cards are loved by the public. They support these formats best.
From FP32 to BF16 and FP16: The Basic Ways to Store Numbers
✦ Format Families
Similar names, different families
First, FP32 is the high-precision baseline. It uses 32 bits, that is 4 bytes, per number to express it finely. If a model's weights are FP32, it's large, but the loss from quantization is almost nil, making it a good reference model.
FP16 and BF16 are both 16 bits, so each number takes 2 bytes. But their internal allocation differs. FP16 can express the fractional part more finely, but the range of numbers it can represent is relatively narrow.
BF16 keeps a wide range similar to FP32 but gives up some fractional fineness. So even though both are 16 bits, they're not the same format, and the choice depends on what work you're doing. That's why it's good to try both for each model.
You test which gives values closer to the result you want. It's hard to say clearly that one is superior.
INT8 and FP8 are 8 bits, roughly 1 byte each. INT8 approximates the original weights by combining a few integer steps with a scale value, and FP8 uses a very small floating-point number system.
FP8 has formats like E4M3, which expresses the fractional part a bit more finely, and E5M2, which handles a wider range of values. Even though both names contain '8,' INT8 and FP8 differ right down to how they hold numbers.
There's also MXFP8. It doesn't just use FP8 numbers; it's a block-based method that attaches a separate scale to every 32 consecutive values. Instead of forcing a single scale onto a whole chunk, it uses a fitting scale for each small section.
Unlike plain FP8, which ends at 8 bits per number, it also needs scale information, but it can better represent data whose range of values varies from section to section.
NVFP4 is, as the name says, a 4-bit floating-point format. It attaches a block scale to every 16 values and also uses an overall scale, storing the values themselves small while adjusting the representable range.
It's a format covered in NVIDIA's low-precision computing technology, but Ollama also announced support for running NVFP4 models on its MLX engine for Apple Silicon.
Still, with these FP-family formats you have to look at the model file, the running engine, and hardware support together. It won't do to assume from the name alone that you can use them as-is in every Ollama environment.
However, the quantization methods mentioned above are inconvenient to use unless you're an NVIDIA graphics card user. So users with AMD or Intel graphics cards, or Macs, end up looking for GGUF quantization methods with "Q" in the name.
But GGUF quantization is, by default, a bit slow. Of course, even some NVIDIA users with too little VRAM use GGUF, which runs well on small VRAM, but you can't help losing out on speed.
If methods like FP16 or BF16 feel like running an executable directly on Windows, GGUF feels like running an executable inside a zip file. When you run them and compare directly, it feels just like that.
Mac users sometimes look for the MLX quantization method optimized for Mac, but since it isn't a universal method, quite often the model you want to use has no MLX version. This is why NVIDIA holds such a strong position in the AI market.
NVIDIA graphics card users just run things assuming any quantization method will be supported, but AMD, Intel, and Mac users have the inconvenience of first having to check "Is this quantization method supported on my graphics card?"
Q8_0, Q4_K_M, Q2_0: Names You See Often in Ollama
✦ Format Traits at a Glance
From FP families to GGUF
Now let's look at Q8_0, Q4_K_M, and Q2_0, which you'll often see when looking for models in Ollama. The Q here points to a family of specific quantization methods used in GGUF.
You shouldn't see them as the same kind as INT8 or FP8 explained earlier just because the numbers can be compared.
Q8_0 is a GGUF quantization method that attaches a per-block scale to 8-bit values. Because it packs 32 weights into 34 bytes, including the scale it takes an average of about 8.5 bits.
The file is fairly large, but it's the choice for keeping the original model's performance better than lower-bit quantization.
Q4_K_M is a compromise widely used for local language models. It's based on K-family block quantization, with a mixed configuration that gives some weights higher precision.
So even though the name says Q4, the actual average is roughly 4.8 to 4.9 bits per weight, depending on the model and implementation. The name Q4 doesn't mean every weight is stored at exactly 4 bits.
This is why people who want to balance file size and answer quality often pick Q4_K_M.
Q2_0, which Ollama just added support for, is bolder. It stores 64 weights in an 18-byte block, so the average including scale is 2.25 bits per weight.
It can pack the same number of weights much smaller than Q4_K_M, but the number of steps that can represent the original numbers also drops sharply.
The chance rises that answers fall apart in long reasoning or coding, or that the model's distinctive abilities get damaged more. Q2_0, Q2_K, and IQ2 are different quantization methods even though the names look alike.
If you're unsure, try Q8_0, Q4_K_M, and Q2_0 first.
The Differences Show Up in the Numbers
✦ Approximate Size for 1 Billion Weights
Weight data only, excluding metadata and non-weight tensors
If we assume 1 billion weights, FP32 needs about 4GB for the weight data alone, FP16 and BF16 about 2GB, and INT8 and FP8 roughly 1GB.
GGUF Q8_0 comes to about 1.06GB because of the scale overhead, Q4_K_M about 0.61GB based on average bit count, and Q2_0 about 0.28GB.
An actual model file also contains metadata and some tensors stored in other formats, so the final file won't exactly match these numbers.
Looking at these numbers alone, Q2_0 might seem like the clear winner. But cutting the size to a quarter doesn't mean you got a free result that preserves intelligence as is.
Quantization is a technology for reducing model size, not magic that removes every cost while keeping the model's ability at the same level.
If Your GPU Isn't NVIDIA... Which Format Should You Pick?
✦ Format Selection Guide
For general local inference
For general local inference, trying Q4_K_M first is a good starting point. You can expect usable quality while lowering memory use.
If you have plenty of memory and answer quality matters more, Q5_K_M or Q6_K are candidates, and so is Q8_0 if you can accept a bigger file. Conversely, if the model doesn't fit in memory and you can't even run it, you can try stronger quantization like Q2_0.
In that case, you should compare quality too, as much as the size shrinks.
This is the meaning of this Ollama update too. Honestly, Q2_0 isn't the default anyone should pick first. It's an option like a small backpack you pull out when you have to run a big model locally somehow.
It's wiser to match quality and size with Q4_K_M or a higher format normally, and try Q2_0 when you're truly short on memory.
That's all for this post. See you next time.
- Nano Banana 2.1: 4K Images at $0.113 Each, Worth It?
- FLUX 3 Image Klein vs Qwen Image 2.1: Preserve or Blend?
- MOSS-Image Arrives in ComfyUI, and Commercial Use Is OK
🌐 Korean version: 10월 16일, Ollama의 Q2 양자화 형식 지원 시작! 양자화가 뭔지 쉽게 알아보자
No comments:
No comments yet. Want to leave the first one?
Post a Comment