Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.
Code4Scene Text-to-SceneImage-to-Scene
20 scenes · score out of 10075 scenes · score out of 100
Swipe to view every configuration →
Sorted by score. Bar labels are scores ×100, rounded down; the table truncates to three decimal places. One color per model provider.
Sorted by score. Bar labels round scores ×100; the table preserves three decimal places. One color per model provider.
Code4Scene Pareto Frontier
Quality against the median cost of one case · Text-to-Scene, 20 scenes
Quality against the cost of one case · Image-to-Scene, 75 public cases
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the median USD per case. Kimi K3.1 Pro (max) is sold by subscription; cost = the token counts its CLI logs, at Kimi K3.1 Pro's OpenRouter list price ($2 input, $0.20 cached input, $10 output per million). Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price. Muse Spark 1.3 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Spark 1.3's per-token list price. Muse Wonder 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at its per-token list price. Qwen3.8 27B (thinking on) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.04 input, $1.35 output per million). Gemma 4 31B (thinking off) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). Gemma 4 31B (thinking on) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). Qwen3.8 27B (thinking off) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.24 input, $2.2 output per million).
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case.
-
29 configurations
Ranking
Research Track, 20 scenes, re-run in October 2026 with every agent in the same sandbox. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical; a case without a valid scene scores 0. Detailed, Overview and Physical are means of the per-case scores. Cost is the median per case. *Kimi K3.1 Pro (max) is sold by subscription; cost = the token counts its CLI logs, at Kimi K3.1 Pro's OpenRouter list price ($2 input, $0.20 cached input, $10 output per million). *Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price. *Muse Spark 1.3 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Spark 1.3's per-token list price. *Muse Wonder 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at its per-token list price. *Qwen3.8 27B (thinking on) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.04 input, $1.35 output per million). *Gemma 4 31B (thinking off) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). *Gemma 4 31B (thinking on) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). *Qwen3.8 27B (thinking off) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.24 input, $2.2 output per million).
75 public cases (25 indoor, 50 outdoor). Case score = 0.8 × Repair F1 + 0.2 × Physical. Repair F1 counts target actors restored to the withheld ground truth.
Research Track, 20 scenes, re-run in October 2026 with every agent in the same sandbox. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical; a case without a valid scene scores 0. Detailed, Overview and Physical are means of the per-case scores. Cost is the median per case. *Kimi K3.1 Pro (max) is sold by subscription; cost = the token counts its CLI logs, at Kimi K3.1 Pro's OpenRouter list price ($2 input, $0.20 cached input, $10 output per million). *Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price. *Muse Spark 1.3 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Spark 1.3's per-token list price. *Muse Wonder 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at its per-token list price. *Qwen3.8 27B (thinking on) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.04 input, $1.35 output per million). *Gemma 4 31B (thinking off) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). *Gemma 4 31B (thinking on) is self-hosted; cost = the token counts its agent logs, at Gemma 4 31B's OpenRouter list price ($0.20 input, $0.40 output per million). *Qwen3.8 27B (thinking off) is self-hosted; cost = the token counts its agent logs, at Qwen3.8 27B's OpenRouter list price ($0.24 input, $2.2 output per million).
75 public cases (25 indoor, 50 outdoor). Case score = 0.8 × Repair F1 + 0.2 × Physical. Repair F1 counts target actors restored to the withheld ground truth.