- Shoonya β Mechanistically Instrumented Compact Code & Graph-Query Models
Shoonya β Mechanistically Instrumented Compact Code & Graph-Query Models
Shoonya (Sanskrit: ΰ€Άΰ₯ΰ€¨ΰ₯ΰ€― β zero / the foundational origin) is a family of from-scratch, open-weights compact code and graph-query language models (17M to 317M parameters, scaling toward 1.3B) developed by VaidhyaMegha Private Limited / Samyama Research. Every checkpoint is trained and audited against live random-matrix singular value spectra (HTSR Ξ±, Von Neumann eRank), execution-guided reinforcement learning (Step-GRPO), and strict SHA-256 benchmark decontamination across Python, Rust, and Samyama OpenCypher.
Key Research & Systems Insights
1,000xβ15,000xSample Efficiency (0.26BVerified Tokens Outperforming39Bβ4,000B350Mβ400MBaselines): Trained from scratch on226.6MAST/compiler/graph-verified pretraining tokens (~0.26Btotal incl. long-context & post-training),Shoonya-Code-350M(317.3Mparams) reaches28.0%(72/257) greedy /28.4%(73/257)Search-TTC/31.9%pass@2on zero-shotMBPP-Sanitizedand15.2%greedy /17.1%β18.3%(28β30/164)Search-TTConOpenAI HumanEvalβsurpassingPolyCoder-400M(39B),CodeGen-350M-mono(264B), andSmolLM2-360M(4,000B) while natively compiling Rust (25.0%β33.3%rustc --testv1, up to73.4%Rustc-64compile) and executingSamyamaOpenCypher(25.0%t=0v1,4.0%β5.0%Cypher-100,99%β100%valid syntax).- Eliminating the Small-Model Vocabulary Tax & Indentation Errors (
0 / 421): Decomposing tied embeddings $E \in \mathbb{R}^{V \times d}$ into rank-r = 256factors ($E_{\text{tok}} \in \mathbb{R}^{V \times 256}$, $P_{\text{proj}} \in \mathbb{R}^{256 \times 1024}$) cuts vocabulary parameters by74.5%(51.5M$\to$13.1M), dedicating95.9%of model weights to20SwiGLUreasoning layers. Pairing this with atomic<|indent_4|>,<|indent_8|>,<|indent_12|>BPE tokens (V = 50,264) compresses indented Python by37.9%and drives out-of-distributionIndentationErrors to0 / 421(0.0%) acrossMBPP+HumanEval. - Closed-Loop Random-Matrix Spectral Certificates (
20 / 20Collapse-Free Layers,4.75 msatd = 2,048): Instead of monitoring cross-entropy loss alone, Shoonya tracks Heavy-Tailed Self-Regularization power-law exponents (HTSR Ξ±), Von Neumann effective rank (eRank), and differentiable Jensen-Shannon head diversity across rotating layers via exact Rust Cyclic Jacobi and Halko-Martinsson-Tropp randomized subspace SVD (4.75 msatStage-32,048 x 2,048;Mean HTSR Ξ± = 2.006,Mean eRank = 78.0%,20/20 Ideal_HeavyTailed_SOTA). - Positive Tri-Modal RLVR (
Step-GRPO+KnOTS) & Active Code/Graph Grounding (FLARE+<|search_query|>): Domain-specializedStep-GRPOLoRAadapters merged in a shared singular basis (KnOTS+TIES) cutHallucinatedIdentifierOrImportby-98.2%and doubledRustc-Exec(16.7%$\to$33.3%). Adding zero-recompute incremental KV<|search_query|>...<|search_result|>pause-and-resume,FLAREShannon-entropy gating, and staticOpenCypherschema-binding verification (v0.8.0-Grounded) liftedMBPP-Sanitizedto28.4%(73/257) and unlocked16.7%(2/12t=0, up to33.3%stochasticN=4) liveSamyamaproperty-graph execution. - Graph-Native Zero-Python Serving (
13.68xTTFTSpeedup,8KYaRN/Dynamic-NTK&-87.5%Cross-Layer KV): Pinning precomputedGQACopy-on-Write Paged KV blocks (PAGE_TOKENS = 16) per(tenant_id, schema_sha256)reducesOpenCypherTime-To-First-Token from7.54s$\to$0.55s(13.68x) with0.0token drift and26.1 tok/s(BF16) /37.5 tok/s(INT8) 8-thread CPU decoding, whileDynamic-NTK/YaRN(mscale) extrapolation to8,192tokens +2:1Cross-Layer KV Sharing cuts8KKV RAM by-87.5%(BF16,80 MiB) /-93.75%(INT8,40 MiB) vs. standardMHA.
1. Architecture Summary & Checkpoint Guide
All current checkpoints use Pre-RMSNorm, Grouped-Query Attention (GQA), half-split RoPE ($\theta = 10{,}000$), SwiGLU, and the 50,264-token CodeTokenizer (50,257 GPT-2 BPE tokens + 4 disjoint FIM/pad sentinels 50257..50260 + 3 atomic indentation tokens 50261..50263):
| Model Variant | Exact Params | $d_{\text{model}}$ | Layers ($L$) | Heads ($H_q / H_{kv}$) | FFN ($d_{ff}$) | Tied Embedding Layout | Context | Primary Artifact in Repo |
|---|---|---|---|---|---|---|---|---|
Shoonya-Code-350M (v0.8.5-OrthoKnOTS, Recommended) |
317,310,976 |
1,024 |
20 |
16 / 4 |
4,096 |
Rank-256 Factorized (-74.5%) |
2,048 |
shoonya_stage2_350m_v085_orthoknots.safetensors (605.3 MiB BF16) |
Shoonya-Code-350M (v0.8.0-Grounded) |
317,310,976 |
1,024 |
20 |
16 / 4 |
4,096 |
Rank-256 Factorized |
2,048 |
shoonya_stage2_350m_v080_grounded.safetensors (605.3 MiB BF16) |
Shoonya-Code-350M (v0.7.5-RLVR) |
317,310,976 |
1,024 |
20 |
16 / 4 |
4,096 |
Rank-256 Factorized |
2,048 |
shoonya_stage2_350m_v075_rlvr.safetensors (605.3 MiB BF16) |
Shoonya-Code-350M (v0.7.0-SFT / Base) |
317,310,976 |
1,024 |
20 |
16 / 4 |
4,096 |
Rank-256 Factorized |
2,048 |
shoonya_stage2_350m_v07_sft.pt / ..._v07_base.pt |
Shoonya-Code-135M (Stage 1) |
142,456,320 |
768 |
12 |
12 / 4 |
3,072 |
Unfactorized Tied | 512 |
shoonya_stage1_135m.pt (543.5 MiB) |
Shoonya-v0.4.0-Small |
16,805,376 |
256 |
4 |
4 / 2 |
1,024 |
Unfactorized Tied | 512 |
shoonya_v0_4_code_small.pt (64.1 MiB) |
(Legacy bootstrap files shoonya_model_v0_1*.pt, pytorch_model.bin, model.onnx, and shoonya_stage2_350m.pt [v0.5.0] are retained strictly for longitudinal reproducibility.)
2. Evaluation, Sub-500M Standing & Mechanistic Health
Sub-500M External Reference Comparison (Zero-Shot pass@1)
For the full longitudinal ablation holding 317,310,976 parameters constant from v0.5.0 (0.43M tokens) $\to$ v0.6.0 (42.66M) $\to$ v0.7.0 (226.60M) $\to$ v0.7.5-RLVR $\to$ v0.8.0-Grounded $\to$ v0.8.5-OrthoKnOTS (plus Rustc-Exec-64, Cypher-Bench-100, and the 6-arm Iso-FLOP Architecture & WSD Wind-Tunnel), see benchmarks/SCALING_V050_V060_V070_SCORECARD.md, benchmarks/V075_VS_V080_COMPARISON_NOTES.md, benchmarks/eval_v085_full593_summary.json, benchmarks/iso_flop_arch_wsd_ablation_v09.json, benchmarks/spectral_certificate_v085_stage2.json, and benchmarks/stage3_telemetry_and_long_context_v081.json.
| Model | Params | Pretraining Tokens | OpenAI HumanEval (164) |
MBPP (pass@1) |
Native Rust & OpenCypher + Spectral Health |
|---|---|---|---|---|---|
PolyCoder-400M (CMU) |
400M |
39B |
2.96% |
β | Multilingual code base model |
SmolLM2-360M-Base (Hugging Face) |
360M |
4,000B |
~0%β12.0% |
19.4% (3-shot) |
General web + code base model |
CodeGen-350M-mono (Salesforce) |
350M |
~264B (+71.7B Py) |
12.76% |
23.11% |
Python-specialized (BigPython) |
SmolLM2-360M-Instruct (Hugging Face) |
360M |
4,000B + SFT/DPO |
~14.2%β14.6% |
Mid-teens | General instruction-tuned SLM |
Shoonya-Code-350M (v0.7.5-RLVR, Greedy t=0) (Ours) |
317.3M |
0.26B |
15.2% (16.5% pass@2) |
27.2% (31.9% pass@2) |
25.0% rustc v1 (9.4% Rustc-64); 0.0% t=0 Cypher; 20/20 Ideal |
Shoonya-Code-350M (v0.8.0-Grounded, Greedy t=0) (Ours) |
317.3M |
0.26B |
15.2% (16.5% pass@2) |
27.6% (31.5% pass@2) |
16.7% rustc v1 (39.1% Rustc-64 compile); 16.7% v1 / 2.0% Cypher-100; 20/20 Ideal |
Shoonya-Code-350M (v0.8.5-OrthoKnOTS, Greedy t=0) (Ours) |
317.3M |
0.26B |
15.2% (25/164) |
28.0% (72/257 β Peak t=0) |
50.0% Rustc-64 compile; 25.0% v1 / 4.0% Cypher-100 (optional_match=8.3%); 20/20 Ideal |
Shoonya-Code-350M (v0.7.5 + Verifier-TTC, N=4) (Ours) |
317.3M |
0.26B |
16.5% (Cons@k=45.3%) |
26.8% (Cons@k=34.6%) |
33.3% rustc v1 (75% compile); 8.3% live Samyama KG (100% valid, 0 errors) |
Shoonya-Code-350M (v0.8.0 + Search-TTC, N=4) (Ours) |
317.3M |
0.26B |
15.9% (Cons@k=46.8%) |
28.4% (73/257 β Peak TTC) |
25.0% rustc v1; 16.7% v1 / 3.0% Cypher-100 (96% valid); 20/20 Ideal |
Shoonya-Code-350M (v0.8.5 + Search-TTC, N=4) (Ours) |
317.3M |
0.26B |
17.1%β18.3% (28β30/164 β Peak) |
28.0% (72/257, 0 syn/ind) |
67.2%β73.4% Rustc-64 compile; 25.0% v1 / 4.0%β5.0% Cypher-100 (99% valid); 20/20 Ideal |
Qwen2.5-Coder-0.5B-Base (Alibaba) |
490M |
5,500B |
28.0% |
23.8% |
Multi-trillion-token code base model |
Qwen2.5-Coder-0.5B-Instruct (Alibaba) |
490M |
5,500B + Heavy SFT/DPO |
61.6% |
57.3% |
Current sub-500M post-trained SOTA ceiling |
(Note: Shoonya evaluates zero-shot MBPP-Sanitized [257 Google-verified test problems, 0 leaks] and OpenAI HumanEval [164 problems]; Stage-A wind-tunnel reaches 93.8%β96.9% pass@1 / 100.0% pass@2, Expected Calibration Error ECE = 0.0410 on MBPP and 0.0628 on HumanEval.)
Pure-Rust Serving & Schema Prefix KV Benchmarks (benchmarks/shoonya_infer_v075_benchmark.json)
Verified on shoonya_stage2_350m_v075_rlvr.safetensors (FP32 logit parity vs. PyTorch max_abs_err = 5.72e-6 on 17.2M/142.5M and 1.29e-5 on 317.3M) across an 8-thread x86-64 CPU (207-token Samyama OpenCypher prompt with a 192-token graph schema prefix):
Precision Mode (317.3M, L = 20) |
Load Time | Cold TTFT (207 Toks) |
Warm Cached TTFT (192 Saved Toks) |
Prefix Cache Speedup | Decode TPOT |
CPU Throughput | Cached Token Drift |
|---|---|---|---|---|---|---|---|
BF16 Weights + FP32 Paged KV |
383.8 ms |
7,543.3 ms |
551.5 ms |
13.68x |
38.35 ms |
26.1 tok/s |
0.0 (Exact) |
INT8 Weights + INT8 Paged KV |
1,124.2 ms |
4,910.1 ms |
421.0 ms |
11.66x |
26.65 ms |
37.5 tok/s |
0.0 (Exact) |
3. Training Corpus & Zero-Leakage Governance
Full shard hashes, source breakdowns, and per-dataset licenses are documented in DATASET_CARD.md, DATA-LICENSES.md, data/shards/v0.9_clean/MANIFEST.json, and historical data/shards/v0.7/MANIFEST.json:
v0.9.0-cleanCommercial-Clean & Graph-Rich Shards (355.50Muint16tokens across36train +1val shard,1,034,482unique docs,0% NC-SA/0% OpenRAIL):75.20%Python (nvidia/OpenCodeInstructexecution-verifiedavg_test_score >= 0.9,OpenCoder opc-sft-stage1/2,CodeFeedback,Magicoder-OSS-75K, full Python3.12stdlib,mbpp:train),17.84%Rust (Apache-2.0Fortytwo-Network/Strandset-Rust-v1,Tesslate/Rust_Dataset,rustc-verified items, first-party Rust crates), and6.96%SamyamaOpenCypher(53,513unique docs,46,005live-executed across all 6 structural query families inSamyamaClient.embedded(),3xrepeat), plus60,000clean SFT and22,000clean RLVR prompts (0leaks across all593protected problems).v0.7.0βv0.8.0Checkpoint Lineage (226.60Muint16tokens +33.55M1024-ctx tokens):63.45%Python,29.03%Rust, and7.52%SamyamaOpenCypher, followed by4,824completion-masked SFT pairs (v0.7.0-SFT),2,848live-verifiedStep-GRPOtasks merged viaKnOTSSVD +TIES(v0.7.5-RLVR), and650live-verified<|search_query|>...<|search_result|>Masked-SFT trajectories (v0.8.0-Grounded).- Cryptographic Decontamination (
0Leaks Across593Held-Out Tasks): Every pretraining, SFT, RLVR, and active-search document passes normalized docstring/question SHA-256 and canonicalized AST/Rust/OpenCypher SHA-256 filtering againstStage-A(8),MBPP-Sanitized(257),OpenAI HumanEval(164),Rustc-Exec-64(64), andCypher-Bench-100(100).
4. Quickstart Usage (Inspect Published .safetensors Weights)
The snippet below runs with standard huggingface_hub, safetensors, and torch (no internal runtime packages required):
import torch
from huggingface_hub import hf_hub_download
from safetensors import safe_open
ckpt_path = hf_hub_download(
repo_id="VaidhyaMegha/Shoonya",
filename="shoonya_stage2_350m_v080_grounded.safetensors",
)
with safe_open(ckpt_path, framework="pt", device="cpu") as f:
meta = f.metadata()
print("Architecture metadata:", meta)
print("Stored tensors:", len(f.keys())) # 264 deduplicated BF16 tensors
# Reconstruct effective [50264, 1024] tied token embeddings from rank-256 factors:
e_tok = f.get_tensor("embeddings.word_embeddings.weight").float() # [50264, 256]
p_proj = f.get_tensor("embeddings.word_embed_proj.weight").float() # [1024, 256]
effective_embed = e_tok @ p_proj.T # [50264, 1024]
print("Effective tied embedding matrix:", tuple(effective_embed.shape))
5. Intended Use, Limitations & License
- Intended Use: Mechanistic interpretability research (
HTSRheavy-tailed weight spectra,eRankdynamics, singular-subspaceLoRAmerging), sample-efficient code/RLVR & active-grounding research, and sandboxed Python, Rust, andSamyama GraphOpenCyphercompletion prototypes. - Limitations & Out-of-Scope Boundaries: At
0.26Bpretraining tokens (317.3Mparams), syntactic and identifier grounding errors are near zero (0.0%indentation,1.2%β1.6%hallucinated identifiers,100%validOpenCypher), and~61%β69%of remaining misses onMBPP/HumanEvalare pureSemanticAssertionFailures reflecting multi-branch algorithmic breadth (targeted next by1Bβ3Bcommercial-clean pretraining andStage 31.3Bdistillation). Models are not instruction- or safety-aligned for open-ended chat and generated code must only be executed inside resource-limited sandboxes. Note also the~17.5%CC-BY-NC-SA-4.0upstream instructional share disclosed inDATA-LICENSES.md. - Licensing Split:
- Published Model Checkpoints & JSON Configs: Apache License 2.0 (
LICENSE) - Documentation, Benchmark Scorecards, Spectral Certificates &
MANIFEST.json:CC-BY-4.0 - Training Stack, Harness & Rust Crates (
shoonya,shoonya_rs,shoonya-infer,shoonya-search): Proprietary / Internal to VaidhyaMegha Private Limited / Samyama.ai (unreleased β not in the public domain or open-sourced)
- Published Model Checkpoints & JSON Configs: Apache License 2.0 (
@software{shoonya2026vaidhyamegha,
title = {Shoonya: Mechanistically Instrumented Compact Code and Graph-Query Models with Spectral Telemetry},
author = {Kunkunuru, Sandeep and {VaidhyaMegha Research}},
year = {2026},
publisher = {VaidhyaMegha Private Limited / Samyama.ai},
url = {https://huggingface.co/VaidhyaMegha/Shoonya}
}
- Downloads last month
- 320