Shoonya β€” Mechanistically Instrumented Compact Code & Graph-Query Models

Shoonya (Sanskrit: ΰ€Άΰ₯‚ΰ€¨ΰ₯ΰ€― β€” zero / the foundational origin) is a family of from-scratch, open-weights compact code and graph-query language models (17M to 317M parameters, scaling toward 1.3B) developed by VaidhyaMegha Private Limited / Samyama Research. Every checkpoint is trained and audited against live random-matrix singular value spectra (HTSR Ξ±, Von Neumann eRank), execution-guided reinforcement learning (Step-GRPO), and strict SHA-256 benchmark decontamination across Python, Rust, and Samyama OpenCypher.


Key Research & Systems Insights

  1. 1,000x–15,000x Sample Efficiency (0.26B Verified Tokens Outperforming 39B–4,000B 350M–400M Baselines): Trained from scratch on 226.6M AST/compiler/graph-verified pretraining tokens (~0.26B total incl. long-context & post-training), Shoonya-Code-350M (317.3M params) reaches 28.0% (72/257) greedy / 28.4% (73/257) Search-TTC / 31.9% pass@2 on zero-shot MBPP-Sanitized and 15.2% greedy / 17.1%–18.3% (28–30/164) Search-TTC on OpenAI HumanEvalβ€”surpassing PolyCoder-400M (39B), CodeGen-350M-mono (264B), and SmolLM2-360M (4,000B) while natively compiling Rust (25.0%–33.3% rustc --test v1, up to 73.4% Rustc-64 compile) and executing Samyama OpenCypher (25.0% t=0 v1, 4.0%–5.0% Cypher-100, 99%–100% valid syntax).
  2. Eliminating the Small-Model Vocabulary Tax & Indentation Errors (0 / 421): Decomposing tied embeddings $E \in \mathbb{R}^{V \times d}$ into rank-r = 256 factors ($E_{\text{tok}} \in \mathbb{R}^{V \times 256}$, $P_{\text{proj}} \in \mathbb{R}^{256 \times 1024}$) cuts vocabulary parameters by 74.5% (51.5M $\to$ 13.1M), dedicating 95.9% of model weights to 20 SwiGLU reasoning layers. Pairing this with atomic <|indent_4|>, <|indent_8|>, <|indent_12|> BPE tokens (V = 50,264) compresses indented Python by 37.9% and drives out-of-distribution IndentationErrors to 0 / 421 (0.0%) across MBPP + HumanEval.
  3. Closed-Loop Random-Matrix Spectral Certificates (20 / 20 Collapse-Free Layers, 4.75 ms at d = 2,048): Instead of monitoring cross-entropy loss alone, Shoonya tracks Heavy-Tailed Self-Regularization power-law exponents (HTSR Ξ±), Von Neumann effective rank (eRank), and differentiable Jensen-Shannon head diversity across rotating layers via exact Rust Cyclic Jacobi and Halko-Martinsson-Tropp randomized subspace SVD (4.75 ms at Stage-3 2,048 x 2,048; Mean HTSR Ξ± = 2.006, Mean eRank = 78.0%, 20/20 Ideal_HeavyTailed_SOTA).
  4. Positive Tri-Modal RLVR (Step-GRPO + KnOTS) & Active Code/Graph Grounding (FLARE + <|search_query|>): Domain-specialized Step-GRPO LoRA adapters merged in a shared singular basis (KnOTS + TIES) cut HallucinatedIdentifierOrImport by -98.2% and doubled Rustc-Exec (16.7% $\to$ 33.3%). Adding zero-recompute incremental KV <|search_query|>...<|search_result|> pause-and-resume, FLARE Shannon-entropy gating, and static OpenCypher schema-binding verification (v0.8.0-Grounded) lifted MBPP-Sanitized to 28.4% (73/257) and unlocked 16.7% (2/12 t=0, up to 33.3% stochastic N=4) live Samyama property-graph execution.
  5. Graph-Native Zero-Python Serving (13.68x TTFT Speedup, 8K YaRN/Dynamic-NTK & -87.5% Cross-Layer KV): Pinning precomputed GQA Copy-on-Write Paged KV blocks (PAGE_TOKENS = 16) per (tenant_id, schema_sha256) reduces OpenCypher Time-To-First-Token from 7.54s $\to$ 0.55s (13.68x) with 0.0 token drift and 26.1 tok/s (BF16) / 37.5 tok/s (INT8) 8-thread CPU decoding, while Dynamic-NTK/YaRN (mscale) extrapolation to 8,192 tokens + 2:1 Cross-Layer KV Sharing cuts 8K KV RAM by -87.5% (BF16, 80 MiB) / -93.75% (INT8, 40 MiB) vs. standard MHA.

1. Architecture Summary & Checkpoint Guide

All current checkpoints use Pre-RMSNorm, Grouped-Query Attention (GQA), half-split RoPE ($\theta = 10{,}000$), SwiGLU, and the 50,264-token CodeTokenizer (50,257 GPT-2 BPE tokens + 4 disjoint FIM/pad sentinels 50257..50260 + 3 atomic indentation tokens 50261..50263):

Model Variant Exact Params $d_{\text{model}}$ Layers ($L$) Heads ($H_q / H_{kv}$) FFN ($d_{ff}$) Tied Embedding Layout Context Primary Artifact in Repo
Shoonya-Code-350M (v0.8.5-OrthoKnOTS, Recommended) 317,310,976 1,024 20 16 / 4 4,096 Rank-256 Factorized (-74.5%) 2,048 shoonya_stage2_350m_v085_orthoknots.safetensors (605.3 MiB BF16)
Shoonya-Code-350M (v0.8.0-Grounded) 317,310,976 1,024 20 16 / 4 4,096 Rank-256 Factorized 2,048 shoonya_stage2_350m_v080_grounded.safetensors (605.3 MiB BF16)
Shoonya-Code-350M (v0.7.5-RLVR) 317,310,976 1,024 20 16 / 4 4,096 Rank-256 Factorized 2,048 shoonya_stage2_350m_v075_rlvr.safetensors (605.3 MiB BF16)
Shoonya-Code-350M (v0.7.0-SFT / Base) 317,310,976 1,024 20 16 / 4 4,096 Rank-256 Factorized 2,048 shoonya_stage2_350m_v07_sft.pt / ..._v07_base.pt
Shoonya-Code-135M (Stage 1) 142,456,320 768 12 12 / 4 3,072 Unfactorized Tied 512 shoonya_stage1_135m.pt (543.5 MiB)
Shoonya-v0.4.0-Small 16,805,376 256 4 4 / 2 1,024 Unfactorized Tied 512 shoonya_v0_4_code_small.pt (64.1 MiB)

(Legacy bootstrap files shoonya_model_v0_1*.pt, pytorch_model.bin, model.onnx, and shoonya_stage2_350m.pt [v0.5.0] are retained strictly for longitudinal reproducibility.)


2. Evaluation, Sub-500M Standing & Mechanistic Health

Sub-500M External Reference Comparison (Zero-Shot pass@1)

For the full longitudinal ablation holding 317,310,976 parameters constant from v0.5.0 (0.43M tokens) $\to$ v0.6.0 (42.66M) $\to$ v0.7.0 (226.60M) $\to$ v0.7.5-RLVR $\to$ v0.8.0-Grounded $\to$ v0.8.5-OrthoKnOTS (plus Rustc-Exec-64, Cypher-Bench-100, and the 6-arm Iso-FLOP Architecture & WSD Wind-Tunnel), see benchmarks/SCALING_V050_V060_V070_SCORECARD.md, benchmarks/V075_VS_V080_COMPARISON_NOTES.md, benchmarks/eval_v085_full593_summary.json, benchmarks/iso_flop_arch_wsd_ablation_v09.json, benchmarks/spectral_certificate_v085_stage2.json, and benchmarks/stage3_telemetry_and_long_context_v081.json.

Model Params Pretraining Tokens OpenAI HumanEval (164) MBPP (pass@1) Native Rust & OpenCypher + Spectral Health
PolyCoder-400M (CMU) 400M 39B 2.96% β€” Multilingual code base model
SmolLM2-360M-Base (Hugging Face) 360M 4,000B ~0%–12.0% 19.4% (3-shot) General web + code base model
CodeGen-350M-mono (Salesforce) 350M ~264B (+71.7B Py) 12.76% 23.11% Python-specialized (BigPython)
SmolLM2-360M-Instruct (Hugging Face) 360M 4,000B + SFT/DPO ~14.2%–14.6% Mid-teens General instruction-tuned SLM
Shoonya-Code-350M (v0.7.5-RLVR, Greedy t=0) (Ours) 317.3M 0.26B 15.2% (16.5% pass@2) 27.2% (31.9% pass@2) 25.0% rustc v1 (9.4% Rustc-64); 0.0% t=0 Cypher; 20/20 Ideal
Shoonya-Code-350M (v0.8.0-Grounded, Greedy t=0) (Ours) 317.3M 0.26B 15.2% (16.5% pass@2) 27.6% (31.5% pass@2) 16.7% rustc v1 (39.1% Rustc-64 compile); 16.7% v1 / 2.0% Cypher-100; 20/20 Ideal
Shoonya-Code-350M (v0.8.5-OrthoKnOTS, Greedy t=0) (Ours) 317.3M 0.26B 15.2% (25/164) 28.0% (72/257 β€” Peak t=0) 50.0% Rustc-64 compile; 25.0% v1 / 4.0% Cypher-100 (optional_match=8.3%); 20/20 Ideal
Shoonya-Code-350M (v0.7.5 + Verifier-TTC, N=4) (Ours) 317.3M 0.26B 16.5% (Cons@k=45.3%) 26.8% (Cons@k=34.6%) 33.3% rustc v1 (75% compile); 8.3% live Samyama KG (100% valid, 0 errors)
Shoonya-Code-350M (v0.8.0 + Search-TTC, N=4) (Ours) 317.3M 0.26B 15.9% (Cons@k=46.8%) 28.4% (73/257 β€” Peak TTC) 25.0% rustc v1; 16.7% v1 / 3.0% Cypher-100 (96% valid); 20/20 Ideal
Shoonya-Code-350M (v0.8.5 + Search-TTC, N=4) (Ours) 317.3M 0.26B 17.1%–18.3% (28–30/164 β€” Peak) 28.0% (72/257, 0 syn/ind) 67.2%–73.4% Rustc-64 compile; 25.0% v1 / 4.0%–5.0% Cypher-100 (99% valid); 20/20 Ideal
Qwen2.5-Coder-0.5B-Base (Alibaba) 490M 5,500B 28.0% 23.8% Multi-trillion-token code base model
Qwen2.5-Coder-0.5B-Instruct (Alibaba) 490M 5,500B + Heavy SFT/DPO 61.6% 57.3% Current sub-500M post-trained SOTA ceiling

(Note: Shoonya evaluates zero-shot MBPP-Sanitized [257 Google-verified test problems, 0 leaks] and OpenAI HumanEval [164 problems]; Stage-A wind-tunnel reaches 93.8%–96.9% pass@1 / 100.0% pass@2, Expected Calibration Error ECE = 0.0410 on MBPP and 0.0628 on HumanEval.)

Pure-Rust Serving & Schema Prefix KV Benchmarks (benchmarks/shoonya_infer_v075_benchmark.json)

Verified on shoonya_stage2_350m_v075_rlvr.safetensors (FP32 logit parity vs. PyTorch max_abs_err = 5.72e-6 on 17.2M/142.5M and 1.29e-5 on 317.3M) across an 8-thread x86-64 CPU (207-token Samyama OpenCypher prompt with a 192-token graph schema prefix):

Precision Mode (317.3M, L = 20) Load Time Cold TTFT (207 Toks) Warm Cached TTFT (192 Saved Toks) Prefix Cache Speedup Decode TPOT CPU Throughput Cached Token Drift
BF16 Weights + FP32 Paged KV 383.8 ms 7,543.3 ms 551.5 ms 13.68x 38.35 ms 26.1 tok/s 0.0 (Exact)
INT8 Weights + INT8 Paged KV 1,124.2 ms 4,910.1 ms 421.0 ms 11.66x 26.65 ms 37.5 tok/s 0.0 (Exact)

3. Training Corpus & Zero-Leakage Governance

Full shard hashes, source breakdowns, and per-dataset licenses are documented in DATASET_CARD.md, DATA-LICENSES.md, data/shards/v0.9_clean/MANIFEST.json, and historical data/shards/v0.7/MANIFEST.json:

  • v0.9.0-clean Commercial-Clean & Graph-Rich Shards (355.50M uint16 tokens across 36 train + 1 val shard, 1,034,482 unique docs, 0% NC-SA / 0% OpenRAIL): 75.20% Python (nvidia/OpenCodeInstruct execution-verified avg_test_score >= 0.9, OpenCoder opc-sft-stage1/2, CodeFeedback, Magicoder-OSS-75K, full Python 3.12 stdlib, mbpp:train), 17.84% Rust (Apache-2.0 Fortytwo-Network/Strandset-Rust-v1, Tesslate/Rust_Dataset, rustc-verified items, first-party Rust crates), and 6.96% Samyama OpenCypher (53,513 unique docs, 46,005 live-executed across all 6 structural query families in SamyamaClient.embedded(), 3x repeat), plus 60,000 clean SFT and 22,000 clean RLVR prompts (0 leaks across all 593 protected problems).
  • v0.7.0–v0.8.0 Checkpoint Lineage (226.60M uint16 tokens + 33.55M 1024-ctx tokens): 63.45% Python, 29.03% Rust, and 7.52% Samyama OpenCypher, followed by 4,824 completion-masked SFT pairs (v0.7.0-SFT), 2,848 live-verified Step-GRPO tasks merged via KnOTS SVD + TIES (v0.7.5-RLVR), and 650 live-verified <|search_query|>...<|search_result|> Masked-SFT trajectories (v0.8.0-Grounded).
  • Cryptographic Decontamination (0 Leaks Across 593 Held-Out Tasks): Every pretraining, SFT, RLVR, and active-search document passes normalized docstring/question SHA-256 and canonicalized AST/Rust/OpenCypher SHA-256 filtering against Stage-A (8), MBPP-Sanitized (257), OpenAI HumanEval (164), Rustc-Exec-64 (64), and Cypher-Bench-100 (100).

4. Quickstart Usage (Inspect Published .safetensors Weights)

The snippet below runs with standard huggingface_hub, safetensors, and torch (no internal runtime packages required):

import torch
from huggingface_hub import hf_hub_download
from safetensors import safe_open

ckpt_path = hf_hub_download(
    repo_id="VaidhyaMegha/Shoonya",
    filename="shoonya_stage2_350m_v080_grounded.safetensors",
)

with safe_open(ckpt_path, framework="pt", device="cpu") as f:
    meta = f.metadata()
    print("Architecture metadata:", meta)
    print("Stored tensors:", len(f.keys()))  # 264 deduplicated BF16 tensors

    # Reconstruct effective [50264, 1024] tied token embeddings from rank-256 factors:
    e_tok = f.get_tensor("embeddings.word_embeddings.weight").float()   # [50264, 256]
    p_proj = f.get_tensor("embeddings.word_embed_proj.weight").float()  # [1024, 256]
    effective_embed = e_tok @ p_proj.T                                  # [50264, 1024]
    print("Effective tied embedding matrix:", tuple(effective_embed.shape))

5. Intended Use, Limitations & License

  • Intended Use: Mechanistic interpretability research (HTSR heavy-tailed weight spectra, eRank dynamics, singular-subspace LoRA merging), sample-efficient code/RLVR & active-grounding research, and sandboxed Python, Rust, and Samyama Graph OpenCypher completion prototypes.
  • Limitations & Out-of-Scope Boundaries: At 0.26B pretraining tokens (317.3M params), syntactic and identifier grounding errors are near zero (0.0% indentation, 1.2%–1.6% hallucinated identifiers, 100% valid OpenCypher), and ~61%–69% of remaining misses on MBPP/HumanEval are pure SemanticAssertionFailures reflecting multi-branch algorithmic breadth (targeted next by 1B–3B commercial-clean pretraining and Stage 3 1.3B distillation). Models are not instruction- or safety-aligned for open-ended chat and generated code must only be executed inside resource-limited sandboxes. Note also the ~17.5% CC-BY-NC-SA-4.0 upstream instructional share disclosed in DATA-LICENSES.md.
  • Licensing Split:
    • Published Model Checkpoints & JSON Configs: Apache License 2.0 (LICENSE)
    • Documentation, Benchmark Scorecards, Spectral Certificates & MANIFEST.json: CC-BY-4.0
    • Training Stack, Harness & Rust Crates (shoonya, shoonya_rs, shoonya-infer, shoonya-search): Proprietary / Internal to VaidhyaMegha Private Limited / Samyama.ai (unreleased β€” not in the public domain or open-sourced)
@software{shoonya2026vaidhyamegha,
  title        = {Shoonya: Mechanistically Instrumented Compact Code and Graph-Query Models with Spectral Telemetry},
  author       = {Kunkunuru, Sandeep and {VaidhyaMegha Research}},
  year         = {2026},
  publisher    = {VaidhyaMegha Private Limited / Samyama.ai},
  url          = {https://huggingface.co/VaidhyaMegha/Shoonya}
}
Downloads last month
320
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support