Post
129
10 Search Subqueries in 200 Microseconds: 1-Bit Consistency Fanout
We built Fanout Diffusion, an ultra-low-latency model that expands a search query into 10 diverse subquery vectors in a single forward pass.
Inspired by the continuous retrieval framework in R4T (arXiv:2603.06397), our goal was to make query expansion fast enough for production search without running language models.
How it works:
• Direct Embedding Space: Takes a query embedded via google/embeddinggemma-300m (768-dim) and predicts 10 distinct subquery vectors simultaneously.
• 1-Step Consistency Denoiser: Generates all 10 slots analytically in a single pass with zero ODE integration loops.
• 1-Bit Hardware Tensor Cores: Quantized ternary weights (-1, 0, +1) running on Ampere/Ada sub-byte PTX instructions with INT4 outer projections.
Key numbers on an RTX 4090:
• Latency: 0.200 ms (ONNX) / 0.329 ms (native C++ CUDA graph)
• Throughput: 417,035 queries/second in batch mode
• Model Size: 1.64 MB (PyTorch QAT) / 32.1 MB (ONNX graph)
• Quality: 0.683 prompt alignment across 540k search queries
Interactive Space:
dejanseo/fanout-diffusion
Model weights, ONNX graph, and C++ engine:
dejanseo/fanout-diffusion
We built Fanout Diffusion, an ultra-low-latency model that expands a search query into 10 diverse subquery vectors in a single forward pass.
Inspired by the continuous retrieval framework in R4T (arXiv:2603.06397), our goal was to make query expansion fast enough for production search without running language models.
How it works:
• Direct Embedding Space: Takes a query embedded via google/embeddinggemma-300m (768-dim) and predicts 10 distinct subquery vectors simultaneously.
• 1-Step Consistency Denoiser: Generates all 10 slots analytically in a single pass with zero ODE integration loops.
• 1-Bit Hardware Tensor Cores: Quantized ternary weights (-1, 0, +1) running on Ampere/Ada sub-byte PTX instructions with INT4 outer projections.
Key numbers on an RTX 4090:
• Latency: 0.200 ms (ONNX) / 0.329 ms (native C++ CUDA graph)
• Throughput: 417,035 queries/second in batch mode
• Model Size: 1.64 MB (PyTorch QAT) / 32.1 MB (ONNX graph)
• Quality: 0.683 prompt alignment across 540k search queries
Interactive Space:
dejanseo/fanout-diffusion
Model weights, ONNX graph, and C++ engine:
dejanseo/fanout-diffusion
- 1 reply
