androidli/laya-multilingual-onnx-int4

laya (multilingual) — ONNX int4 weight-only quantization

4-bit ONNX build of convaiinnovations/laya multilingual checkpoint (322M: ModernBERT-base 22L×768 encoder + 2-layer decision head, vocab 256k), produced with ONNX Runtime's weight-only quantizers.

As far as we can tell this is the first 4-bit ONNX build of laya published anywhere: the existing public quantizations are int8 ONNX, GGUF and CoreML/MLX — no ONNX 4-bit, and nobody has quantized the 256k vocabulary table.

laya is a decision model: it does not generate text; it returns typed answers (choice distribution + confidence / score / noul) for a state + question. Everything here is inference-only weights.

What's inside

file	what	size
laya-multilingual-int4-blk32.onnx (+ .data)	layer weights 4-bit (MatMulNBits, block 32, symmetric) + vocabulary 4-bit (GatherBlockQuantized, block 32)	199 MB
laya-multilingual-int4-layer32-vocab128.onnx (+ .data)	same layer quantisation, vocabulary at block 128 — smallest build	181 MB
tokenizer/, encoder/config.json, rl_agent_config.json	copied verbatim from the base repo, so laya.ONNXAgent() can load this directory standalone	34 MB
scripts/	the exact export / quantize / eval scripts used	small
results/onnx_results.json	per-item accuracy of every build we compared (60 items; no timing fields)	small

Reference point for size: the fp32 multilingual checkpoint is 1,288 MB on disk, of which 786 MB (61%) is the token embedding table [256000, 768].

How it was quantized

1. scripts/export_onnx.py — trace the PyTorch laya.Agent to ONNX (opset 18). The graph has 145 MatMul nodes; the token embedding is a Gather.
2. scripts/int4_quantize.py — onnxruntime.quantization.matmul_nbits_quantizer.MatMulNBitsQuantizer with block_size=32, is_symmetric=True. 96/145 MatMul nodes (the constant-weight projections) become com.microsoft::MatMulNBits; attention QK^T / softmax·V keep fp32 kernels.
3. scripts/emb_quant.py --mode u4-gbq — the vocabulary [256000, 768] becomes com.microsoft::GatherBlockQuantized (packed uint4 + per-block fp32 scales, symmetric zero-point 8). This step is what makes the model small. A plain DequantizeLinear would materialize the whole table as fp32 at runtime, so the saving would be disk-only; GatherBlockQuantized keeps the table packed and dequantizes only the rows actually looked up.

Net effect on the weights: the layer projections go 501 MB → 83 MB, the vocabulary 786 MB → 123 MB.

Quality on our eval (accuracy only)

60 real knowledge-base entries → one choice question over 8 real project labels + 4 plausible decoys; accuracy = the picked label vs. the label the entry carries.

build	accuracy	resident	size
fp32 ONNX (reference)	0.683	1.56 GB	1,288 MB
int8 ONNX (dynamic)	0.600	1.22 GB	310 MB
layers 4-bit blk32 + vocab 4-bit blk32 (this repo)	0.700	1.20 GB	199 MB
layers 4-bit blk32 + vocab 4-bit blk128 (this repo)	0.683	1.14 GB	181 MB

Honest reading: this is our 60-item set, so differences under ~6pp are noise at this n — the 4-bit builds match the fp32 reference on this task rather than beating it. The int8 build (dynamic quantization, a single per-tensor scale for the whole vocabulary) is the one that visibly degrades. Pick blk32 when you want to stay closest to the base model, blk128 when you want the smallest file.

Usage

The tokenizer/config files are included, so this directory is self-contained:

from laya.onnx_agent import ONNXAgent        # note: not re-exported at the laya top level

agent = ONNXAgent("path/to/this/repo", onnx_path="laya-multilingual-int4-blk32.onnx")
out = agent.predict(
    "量化后端点精度下降，根因是激活函数在低位宽下的分布偏移，需要重新校准。",
    {"q": {"type": "choice", "instructions": "这段内容属于哪个主题域？",
           "criteria": {"quant": "量化 低位宽 校准", "deploy": "部署 服务 端口"}}},
)
print(out["answers"]["q"]["choice"], out["answers"]["q"]["confidence"])

Requires pip install laya onnxruntime. The laya package supplies the tokenizer/config plumbing and the typed-answer handling; the ONNX session runs the forward pass. Session options (thread count, graph optimisation level) can be set by constructing the ort.InferenceSession yourself and assigning it to agent.session.

Attribution and changes

• base model: convaiinnovations/laya by Convai Innovations, Apache-2.0. This build is a derivative work and keeps the same licence.
• changes made: weights converted to ONNX (opset 18) and quantized to 4-bit weight-only (MatMulNBits for the linear projections, GatherBlockQuantized for the vocabulary); tokenizer/config files copied verbatim from the base repo.
• the base checkpoint is unchanged and available at the link above.

Built and measured by @greatYiTin.