TurBoQuant

TurBoQuant is a training-free, data-oblivious vector quantization method that achieves near-optimal compression for high-dimensional vectors — the first method to deliver zero quantization overhead at any target bit width. It tackles a fundamental problem in deploying large language models: the KV cache, which stores all previous key-value pairs during autoregressive generation, grows linearly with sequence length and dominates GPU memory at long contexts.

Consider a 70B parameter model (Llama-2-70B) with $L = 80$ layers, $H = 64$ attention heads, head dimension $d_h = 128$, at a 128K context window. The KV cache alone requires $2 \times 80 \times 64 \times 128 \times 131{,}072 \times 2$ bytes (FP16) = ~43 GB — exceeding the model weights themselves on a single GPU. This memory bottleneck limits context length, batch size, and serving throughput.

Traditional quantization methods reduce this by storing values in fewer bits (INT8 = 1 byte, INT4 = 0.5 bytes), but they require quantization constants — scale factors $s$ and zero-points $z$ — stored in full precision (FP32 = 4 bytes) for every block of $B$ values. This overhead adds $32/B$ extra bits per value: at 2-bit quantization with block size 32, the overhead is 1 full bit per value — a 50% penalty that undermines the compression goal. Methods like GPTQ, AWQ, and SmoothQuant all suffer from this overhead because they compute data-dependent statistics that must be stored alongside the compressed data.

TurBoQuant eliminates this overhead entirely through a two-stage approach: PolarQuant converts vectors from Cartesian to polar coordinates after random rotation, exploiting the fact that randomly rotated high-dimensional vectors have angle distributions that are known analytically (Beta distributions) — no per-block constants needed. QJL (Quantized Johnson-Lindenstrauss) then corrects the residual quantization error using only sign bits (+1/-1), the most extreme form of quantization.

The result: 3-bit quantization with zero accuracy loss on LongBench, Needle In A Haystack, ZeroSCROLLS, and RULER benchmarks, achieving 5.3x memory reduction (6x practical) and 8x speedup in attention computation on H100 GPUs using custom JAX kernels. Presented at ICLR 2026, TurBoQuant operates within a constant factor of approximately 2.7x of the Shannon rate-distortion lower bound — provably near-optimal for Gaussian sources at any bit width and any dimension.