Transformers documentation

HIGGS

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

HIGGS

HIGGS applies a Hadamard transform to each weight matrix and then rounds groups of weights to a grid that minimizes the mean squared error, without calibration data. Use it for 2- to 4-bit weights on NVIDIA GPUs, with the FLUTE kernels running the matmuls.

The table below summarizes what HIGGS supports in Transformers.

ConfigHiggsConfig
quant_methodhiggs
WorkflowQuantize while loading
Bits2, 3, 4
HardwareCUDA
Save✅
Fine-tune❌
Auto-compile❌

Install HIGGS

Install FLUTE 0.3.0 or later, fast-hadamard-transform, and Accelerate.

pip install -U "flute-kernel>=0.3.0" fast_hadamard_transform accelerate

Quantize a model with HIGGS

Create a HiggsConfig and pass it to from_pretrained(). Set device_map to your CUDA devices, and load the model in fp16 or bf16.

from transformers import AutoModelForCausalLM, HiggsConfig

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    quantization_config=HiggsConfig(bits=4),
    device_map="cuda",
)

After the weights load, FLUTE tunes its kernels for each layer on your GPU and repacks the weights. The tuning results are stored in the config’s tune_metadata, so a saved model doesn’t need to tune again.

Configure HIGGS

These HiggsConfig parameters change how the weights are quantized.

ParameterDefaultEffect
bits4Number of bits per weight. Supports 2, 3, and 4.
p2Dimension of the quantization grid, either 1 or 2. A grid with p=2 rounds pairs of weights together and is more accurate.
group_size256Number of weights that share a scale. Supports 64, 128, and 256. Must divide hadamard_size.
hadamard_size512Size of the Hadamard transform. Each layer’s input dimension is padded to a multiple of this size. Smaller values reduce accuracy.

Speed up inference with torch.compile

Without compilation, HIGGS runs slower than bf16. generate() doesn’t compile HIGGS models automatically, so compile the model yourself with torch.compile.

model = torch.compile(model)

The table below shows forward passes per second for Llama-3.1-8B-Instruct on an RTX 4090.

Batch sizebf16 (compiled)HIGGS 4-bitHIGGS 4-bit (compiled)
15941124
45742123
165641120

Limitations

  • HIGGS needs a CUDA GPU. Loading without one raises a NotImplementedError.
  • Loading without a device_map, or with one that includes "cpu" or "disk", raises a ValueError.
  • Loading in a dtype other than fp16 or bf16 raises a ValueError.

Resources

Update on GitHub