Why it matters
Serving is often limited by memory bandwidth rather than arithmetic, so representing each weight in fewer bits can raise throughput and fit a larger model on a card. The catch is that mapping floats onto a small integer grid loses information where the distribution has outliers, and those outliers are frequently the weights that matter for precise tasks. Quantization is therefore an accuracy engineering problem, not a compression setting.
Mental model
It is rounding prices to the nearest dollar on a shelf label: most of the time the customer cannot tell, but an item priced at 9.99 becomes a different decision. The analogy breaks because the rounding error compounds across layers, so a per-weight error that looks negligible can flip a token many steps later.
How it works
- 1Choose the scheme first: weight-only for memory, weight-and-activation for throughput, cache quantization for concurrency.
- 2Pick granularity, since per-channel or per-group scales preserve more of the distribution than one tensor-wide scale.
- 3Calibrate on a sample that matches production traffic, including the long inputs and rare formats that stress the range.
- 4Evaluate the quantized model against the float baseline on the labelled slice, not on perplexity alone.
- 5Confirm the engine has a fused kernel for the format; an unsupported path dequantizes and can be slower than the original.
- 6Measure device memory and tokens per second under realistic concurrency before committing the format to the registry.
Step 1 of 6
Choose the scheme first: weight-only for memory, weight-and-activation for throughput, cache quantization for concurrency.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Choose the scheme first: weight-only for memory, weight-and-activation for throughput, ca… 2. Pick granularity, since per-channel or per-group scales preserve more of the distribution… 3. Calibrate on a sample that matches production traffic, including the long inputs and rare… 4. Evaluate the quantized model against the float baseline on the labelled slice, not on per… 5. Confirm the engine has a fused kernel for the format; an unsupported path dequantizes and… 6. Measure device memory and tokens per second under realistic concurrency before committing…
- Choose the scheme first: weight-only for memory, weight-and-activation for throughput, cache quantization for concurrency.
- Pick granularity, since per-channel or per-group scales preserve more of the distribution than one tensor-wide scale.
- Calibrate on a sample that matches production traffic, including the long inputs and rare formats that stress the range.
- Evaluate the quantized model against the float baseline on the labelled slice, not on perplexity alone.
- Confirm the engine has a fused kernel for the format; an unsupported path dequantizes and can be slower than the original.
- Measure device memory and tokens per second under realistic concurrency before committing the format to the registry.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Weight-only four-bit | Large memory savings with modest quality loss | Decode speed depends on kernel support | The engine lacks a fused dequantization kernel |
| Weight and activation eight-bit | Higher throughput on supported hardware | Activation outliers hurt accuracy | Prompts contain values the calibration missed |
| Quantized cache | More concurrent sequences | Attention numerics shift at long context | Tasks need precise recall over very long inputs |
| Per-group scales | Better accuracy at low bit width | Extra metadata and kernel complexity | Memory overhead matters at extreme compression |
Good for
Large memory savings with modest quality loss
Cost
Decode speed depends on kernel support
Fails when
The engine lacks a fused dequantization kernel
- The engine lacks a fused dequantization kernel
- Prompts contain values the calibration missed
- Tasks need precise recall over very long inputs
- Memory overhead matters at extreme compression
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Weight-only four-bit is good for Large memory savings with modest quality loss, costs Decode speed depends on kernel support, and fails when The engine lacks a fused dequantization kernel. Weight and activation eight-bit is good for Higher throughput on supported hardware, costs Activation outliers hurt accuracy, and fails when Prompts contain values the calibration missed. Quantized cache is good for More concurrent sequences, costs Attention numerics shift at long context, and fails when Tasks need precise recall over very long inputs. Per-group scales is good for Better accuracy at low bit width, costs Extra metadata and kernel complexity, and fails when Memory overhead matters at extreme compression
Failure modes
- Accuracy cliff on the hard slice. Aggregate scores hold while code and multi-step reasoning regress. Debug: run the labelled hard slice and diff per-item pass rates against the float baseline.
- Calibration mismatch. Scales fitted on short chat prompts distort long documents with unusual token ranges. Debug: recompute scales on a length-stratified sample and compare the same evaluation.
- Slower after quantizing. The engine falls back to a dequantize-then-multiply path and latency rises. Debug: profile the kernel names in use and confirm a fused path is selected.
In production
Keep a float baseline deployable and record the quantization recipe beside the artefact. Monitor refusal rate, output length and task pass rate after any format change, since the first symptom is often shorter or more cautious answers.
Common misconceptions
People think: Quantization is free compression.
Actually: It removes precision, and the removal shows up unevenly across tasks; the format has to be evaluated, not assumed.
People think: Lower bits always mean faster inference.
Actually: Speed depends on whether the hardware has a fused kernel for the format; without one, unpacking costs more than the bandwidth saved.
Interview angle
Separate weight, activation and cache precision, explain the calibration step, and name the evaluation that would block a rollout. Mention kernel support as the reason a format can be slower. Citing a bit width as though it were a quality guarantee is the weak version.
Sources
- Frantar et al., GPTQ — post-training weight quantization with layer-wise reconstruction.
- Lin et al., AWQ — activation-aware scaling that protects salient weights.
- Hugging Face, Quantization — format families and their engine support.