Quantization of LLMs compresses model weights lossily, and factual knowledge turns out to be the first casualty. Testing 55 quantizations of Qwen3.6 27B using GGUF formats from Unsloth, Bartowski, and llama.cpp against the Incompressible Knowledge Probes (IKP) benchmark reveals a nonlinear degradation pattern. Models quantized to 5-bit or above (>20 GB) retain near-original quality, while 3-bit and 2-bit variants show steep drops — especially for obscure trivia tiers. Crucially, the degradation correlates linearly with KL divergence from the original BF16 model, not with model size alone. The key insight: quantization is blind compression that doesn't optimize for training data, unlike training a smaller model from scratch. Practical takeaways: Q8_0 is a safe bet, Q4_K_M is generally acceptable, and for knowledge-intensive tasks (like navigating a large Python package ecosystem), model size and quantization level matter significantly.
3.6K Impressions1 Comment