Transformers documentation
HIGGS
HIGGS
HIGGS applies a Hadamard transform to each weight matrix and then rounds groups of weights to a grid that minimizes the mean squared error, without calibration data. Use it for 2- to 4-bit weights on NVIDIA GPUs, with the FLUTE kernels running the matmuls.
The table below summarizes what HIGGS supports in Transformers.
| Config | HiggsConfig |
quant_method | higgs |
| Workflow | Quantize while loading |
| Bits | 2, 3, 4 |
| Hardware | CUDA |
| Save | ✅ |
| Fine-tune | ❌ |
| Auto-compile | ❌ |
Install HIGGS
Install FLUTE 0.3.0 or later, fast-hadamard-transform, and Accelerate.
pip install -U "flute-kernel>=0.3.0" fast_hadamard_transform accelerateQuantize a model with HIGGS
Create a HiggsConfig and pass it to from_pretrained(). Set device_map to your CUDA devices, and load the model in fp16 or bf16.
from transformers import AutoModelForCausalLM, HiggsConfig
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-8B",
quantization_config=HiggsConfig(bits=4),
device_map="cuda",
)After the weights load, FLUTE tunes its kernels for each layer on your GPU and repacks the weights. The tuning results are stored in the config’s tune_metadata, so a saved model doesn’t need to tune again.
Configure HIGGS
These HiggsConfig parameters change how the weights are quantized.
| Parameter | Default | Effect |
|---|---|---|
bits | 4 | Number of bits per weight. Supports 2, 3, and 4. |
p | 2 | Dimension of the quantization grid, either 1 or 2. A grid with p=2 rounds pairs of weights together and is more accurate. |
group_size | 256 | Number of weights that share a scale. Supports 64, 128, and 256. Must divide hadamard_size. |
hadamard_size | 512 | Size of the Hadamard transform. Each layer’s input dimension is padded to a multiple of this size. Smaller values reduce accuracy. |
Speed up inference with torch.compile
Without compilation, HIGGS runs slower than bf16. generate() doesn’t compile HIGGS models automatically, so compile the model yourself with torch.compile.
model = torch.compile(model)The table below shows forward passes per second for Llama-3.1-8B-Instruct on an RTX 4090.
| Batch size | bf16 (compiled) | HIGGS 4-bit | HIGGS 4-bit (compiled) |
|---|---|---|---|
| 1 | 59 | 41 | 124 |
| 4 | 57 | 42 | 123 |
| 16 | 56 | 41 | 120 |
Limitations
- HIGGS needs a CUDA GPU. Loading without one raises a
NotImplementedError. - Loading without a
device_map, or with one that includes"cpu"or"disk", raises aValueError. - Loading in a dtype other than fp16 or bf16 raises a
ValueError.
Resources
- Pushing the Limits of Large Language Model Quantization via the Linearity Theorem paper
- FLUTE repository
- HIGGS models collection from ISTA-DASLab