Transformers documentation
HQQ
HQQ
Half-Quadratic Quantization (HQQ) optimizes its quantization parameters from the weights alone, so it doesn’t need calibration data. Use it to quantize a model to as few as 1 or 2 bits while it loads.
The table below summarizes what HQQ supports in Transformers.
| Config | HqqConfig |
quant_method | hqq |
| Workflow | Quantize while loading |
| Bits | 1, 2, 3, 4, 8 |
| Hardware | CUDA, XPU |
| Save | ✅ |
| Fine-tune | ✅ |
| Auto-compile | ❌ |
Install HQQ
Install hqq 0.2.1 or later.
pip install -U hqq
Quantize a model with HQQ
Create a HqqConfig and pass it to from_pretrained(). The example below quantizes every linear layer to 4 bits in groups of 64 weights.
import torch
from transformers import AutoModelForCausalLM, HqqConfig
quantization_config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-0.6B",
quantization_config=quantization_config,
device_map="cuda",
)Configure HQQ
These HqqConfig parameters change how the weights are quantized.
| Parameter | Default | Effect |
|---|---|---|
nbits | 4 | Number of bits per weight. Supports 8, 4, 3, 2, and 1. |
group_size | 64 | Number of weights that share a scale and zero point. Smaller groups are more accurate and use more memory. |
axis | 1 | Axis to group weights along, either 0 or 1. The faster torchao and BitBlas backends work only with 1. |
view_as_float | False | Stores the quantized weights as a float dtype, for distributed training. |
skip_modules | ["lm_head"] | Layer types to leave unquantized. |
Quantize layer types with different settings
Pass dynamic_config to quantize each type of linear layer with its own settings. Each key is a layer tag, the module name with model, layers, and the layer number removed. For example, model.layers.3.self_attn.q_proj has the tag self_attn.q_proj.
The example below quantizes the attention projections to 4 bits and the MLP projections to 3 bits.
q4_config = {"nbits": 4, "group_size": 64}
q3_config = {"nbits": 3, "group_size": 32}
quantization_config = HqqConfig(dynamic_config={
"self_attn.q_proj": q4_config,
"self_attn.k_proj": q4_config,
"self_attn.v_proj": q4_config,
"self_attn.o_proj": q4_config,
"mlp.gate_proj": q3_config,
"mlp.up_proj": q3_config,
"mlp.down_proj": q3_config,
})A tag applies to that layer type in every decoder layer, so you can’t give one decoder layer different settings from the others. Linear layers whose tags aren’t in dynamic_config stay unquantized. The same tags are used to match skip_modules.
Choose a backend
HQQ runs the quantized layers through a backend from the hqq library. The PyTorch and CUDA backends work with PEFT fine-tuning and older GPUs. Switch backends with HQQLinear.set_backend.
from hqq.core.quantize import HQQBackend, HQQLinear
HQQLinear.set_backend(HQQBackend.PYTORCH)For faster 4-bit inference, convert the quantized model to a fused kernel backend, like torchao, after loading it. The torchao backend needs axis=1, which is the default.
from hqq.utils.patching import prepare_for_inference
prepare_for_inference(model, backend="torchao_int4")See the hqq Backend guide for the available backends.
Limitations
- HQQ can’t offload to the CPU or disk. Loading with a
device_mapthat includes"cpu"or"disk"raises aValueError. - Layer tags can’t target one decoder layer, so
dynamic_configandskip_modulesapply to a layer type across the whole model.
Resources
- Half-Quadratic Quantization of Large Machine Learning Models blog post
- hqq repository