diff --git "a/index.html" "b/index.html" --- "a/index.html" +++ "b/index.html" @@ -26,7 +26,7 @@ svg.ch .dim{opacity:.42}svg.ch g.row:hover{opacity:1}svg.ch .hot{fill:var(--c4s-

Research Track · 20 scenes · 29 coding-agent configurations · re-run in October 2026.

Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.

Code4Scene Text-to-Scene

20 scenes · score out of 10075 scenes · score out of 100

Swipe to view every configuration →

Sorted by score. Bar labels are scores ×100, rounded down; the table truncates to three decimal places. One color per model provider.

Code4Scene Pareto Frontier

Quality against the median cost of one case · Text-to-Scene, 20 scenes

Swipe to explore all configurations →

Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the median USD per case. Kimi K3.1 Pro (max) is sold by subscription; cost = the token counts its CLI logs, at Kimi K3.1 Pro's OpenRouter list price ($2 input, $0.20 cached input, $10 output per million). Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price. Muse Spark 1.3 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Spark 1.3's per-token list price. Muse Wonder 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at its per-token list price. Qwen3.8 27B (thinking on) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.04 input, $1.35 output per million). Gemma 4 31B (thinking off) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). Gemma 4 31B (thinking on) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). Qwen3.8 27B (thinking off) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.24 input, $2.2 output per million).

-
29 configurations
Ranking

Research Track, 20 scenes, re-run in October 2026 with every agent in the same sandbox. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical; a case without a valid scene scores 0. Detailed, Overview and Physical are means of the per-case scores. Cost is the median per case. *Kimi K3.1 Pro (max) is sold by subscription; cost = the token counts its CLI logs, at Kimi K3.1 Pro's OpenRouter list price ($2 input, $0.20 cached input, $10 output per million). *Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price. *Muse Spark 1.3 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Spark 1.3's per-token list price. *Muse Wonder 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at its per-token list price. *Qwen3.8 27B (thinking on) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.04 input, $1.35 output per million). *Gemma 4 31B (thinking off) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). *Gemma 4 31B (thinking on) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). *Qwen3.8 27B (thinking off) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.24 input, $2.2 output per million).

#Agent configurationScoreDetailedOverviewPhysicalMedian cost
1Claude Sonnet 5.5 (max)ParetoAnthropic
0.837
0.8030.8120.945$26.93
2Claude Opus 5.5 (max)ParetoAnthropic
0.832
0.7630.8180.943$26.90
3Claude Fable 5.1 (max)ParetoAnthropic
0.800
0.7460.7560.983$23.42
4GPT-6 Astra (max)ParetoOpenAI
0.790
0.7600.8110.759$14.25
5GPT-6.1 Sol (max)ParetoOpenAI
0.784
0.7420.7780.845$2.59
6Gemini 4 Argon (high)Google
0.777
0.6880.7560.930$4.83
7DeepSeek V4.1 Flash 1017 (max)open weightsParetoDeepSeek
0.750
0.6380.7630.824$0.96
8GPT-5.6 Sol (max)OpenAI
0.738
0.6480.7190.886$6.87
9Kimi K3.1 Pro (max)Moonshot AI
0.724
0.6710.7020.844$4.82*
10Claude Haiku 5.5 (max)Anthropic
0.718
0.6520.6670.937$2.43
11Muse Wonder Full 1.0 (max)Meta
0.715
0.6660.6960.819$3.66*
12Grok 4.7 (xhigh)xAI
0.700
0.6910.6860.751$3.98
13Grok 4.6 (xhigh)xAI
0.696
0.6430.7170.689$7.41
14Muse Spark 1.3 (max)Meta
0.696
0.6410.6670.839$3.24*
15Gemini 3.8 Flash (high)Google
0.694
0.5780.6620.905$3.85
16GPT-5.6 Terra (max)OpenAI
0.688
0.6410.6660.801$2.90
17GPT-6 Luna (max)ParetoOpenAI
0.688
0.6060.6660.834$0.16
18GLM-5.3 Flash (max)open weightsZ.ai
0.663
0.6130.6680.699$0.94
19Muse Wonder 1.0 (max)Meta
0.644
0.6090.6200.753$2.17*
20MiMo V2.6 Pro (max)open weightsXiaomi
0.632
0.5720.6150.745$0.92
21GPT-5.6 Luna (max)ParetoOpenAI
0.630
0.5850.6040.750$0.16
22Muse Spark Open (max)open weightsMeta
0.611
0.5830.5910.700$2.01
23Qwen3.8 27B (thinking on)open weightsAlibaba
0.571
0.5080.5240.778$0.90*
24Gemma 4 31B (thinking off)open weightsParetoGoogle
0.553
0.4870.5070.755$0.04*
25Nemotron 3.5 Super VL (max)open weightsNVIDIA
0.520
0.4680.4590.756$0.14
26Gemma 4 31B (thinking on)open weightsGoogle
0.517
0.4480.4690.731$0.06*
27Qwen3.8 27B (thinking off)open weightsAlibaba
0.514
0.5000.4520.716$0.75*
28Step 5 Preview (max)open weightsStepFun
0.503
0.4560.4960.569$3.82
29DeepSeek V4.1 Flash (max)open weightsDeepSeek
0.371
0.3570.3240.526$0.35

75 public cases (25 indoor, 50 outdoor). Case score = 0.8 × Repair F1 + 0.2 × Physical. Repair F1 counts target actors restored to the withheld ground truth.

#Agent configurationScoreRepair F1PhysicalIndoorOutdoorCost / case
1Gemini 3.8 Flash (high)ParetoGoogle
0.581
0.5270.7960.7170.513$1.35
2GPT-6 Astra (max)OpenAI
0.515
0.4450.7960.7850.380$3.32
3Claude Opus 5 (max)Anthropic
0.468
0.3890.7860.6180.393$4.46
4Claude Fable 5.1 (max)Anthropic
0.424
0.3320.7950.5350.369$3.09
5GPT-5.6 Sol (high)OpenAI
0.393
0.3190.6890.4970.341$1.88
6Muse Spark 1.3 (medium)ParetoMeta
0.358
0.2460.8070.6330.221$0.07
7GLM-5.3 Flash (max)open weightsZ.ai
0.320
0.1880.8480.4650.247$0.09
8Qwen 3.8 27B (thinking on)open weightsAlibaba
0.226
0.1180.6600.2970.191$0.26
9DeepSeek V4.1 Flash (high)open weightsParetoDeepSeek
0.210
0.0660.7870.1890.221$0.05
10Qwen 3.8 27B (thinking off)open weightsAlibaba
0.200
0.0830.6670.2660.166$0.28
11