How the arena ranks AI tools for Blender

The arena runs in seasons. A season has a fixed prompt set and a fixed roster, and every competitor in it starts from the Glicko-2 default — no rating carries over between seasons. A tool that finished top of one season begins the next at 1500 ±350 like everyone else.

When a season closes, its standings are frozen and published as a final result that never moves again, and its ballots stay downloadable under their own export. Nothing is deleted at a season boundary: a new season simply starts collecting into a new ballot namespace. That is also why closed seasons keep their original output files at their original paths — the artifact a voter judged has to stay exactly where it was, or the archived result stops being checkable.

A competitor may re-enter a new season with fresh outputs or with the artifacts it already submitted, and which one it did is stated on every output. Re-running the benchmark is not required, but the prompt set version must match.

Glicko-2's rating deviation is the standard error of a rating, so the gap between two competitors has a standard error of √(RD_a² + RD_b²) and a two-sided z-test decides whether the gap is real.

Competitors whose ratings do not separate at p < 0.05 share a placing, shown as a technical draw. A competitor joins a drawn group only if it is inseparable from every member already in it — otherwise a chain of near-ties could drag a clearly weaker entrant into the top group one link at a time.

This is the rule that decided Season 1: the top four were spread over 108 rating points with a standard error near 100 on every pairing, so they share first place rather than being ordered by a difference the ballots cannot support. The p-value for every pairing is published on the Season 1 results page.

Each season, every tool on the roster runs the complete benchmark prompt set under identical rules:

• –The prompt text is used verbatim — no tool-specific tuning.
• –Default settings for each tool; no expert parameter tweaking.
• –One generation per prompt, with a single retry allowed only on a technical failure (crash, timeout, empty output).
• –No manual cleanup, retopology, or material edits afterward.
• –Outputs are exported to a common format (GLB) and rendered with identical lighting for comparison.
• –Three generation paths are in play, and the one used is noted on every single output: procedural three.js code runs, native Blender agentic sessions working on a live scene, and one-shot Blender Python where the model writes a single bpy script executed once headlessly. Mesh generators produce meshes from the same prompt text. The arena compares the final output, whatever the path to it — which means it is partly comparing approaches, not only models. Read the standings with that in mind.
• –Model families compete per-model: Claude Haiku 4.5, Sonnet 5, Opus 4.8, and Fable 5 each generated their Season 1 entries independently — separate sessions, no shared code, no access to each other's outputs.
• –A product entrant is benchmarked as the product, not as the model inside it. 3D-Agent is listed under its own name across seasons even though the models it runs on have changed, because what the arena is scoring is the output a user actually gets. When its pipeline changes materially, that is recorded as a new season entry rather than a silent swap of old files.

Known deviation, Round 2. GPT-5.6 Sol ran under an explicitly enforced read-only sandbox. Claude Opus 5 was given the same written constraints — no file access, no downloads, no imported assets — but its run was not sandbox-enforced, because the flag that restricts its tooling conflicted with how the prompt was passed. Both models' raw responses were pure Python with no tool calls (the returned text and the extracted script differ by a single trailing newline in every case), so we have no evidence the difference affected any output. It is recorded here because an unenforced constraint is not the same as an enforced one, and the harness has since been changed so every future entrant runs under the same restriction.

• –Voters see two outputs generated from the same prompt, side by side, in identical viewers.
• –Tool names are hidden until after the vote. Left/right placement is randomized.
• –Four ballot options: A is better, B is better, Tie, or Both Bad.
• –One ballot per matchup, per voter, per day is counted. Extra ballots are still stored — nothing is deleted — but excluded from the ratings.

Every ballot is written to durable object storage the moment it is cast, one record per vote: timestamp, prompt id, both competitors, the ballot, and a voter key. Standings are recomputed from that record set — there is no hand-maintained score anywhere in the system, and no vote total is ever typed in by a human.

The voter key is a one-way hash of IP address, user agent and the current date. It rotates daily, so it can't be used to follow anyone over time, but it is stable enough within a day to detect ballot stuffing.

The full dataset is public and licensed CC BY 4.0 — the raw ballots (CSV) and the computed standings (JSON). Anyone can recompute the Glicko-2 numbers from the ballots and get the same answer. If they don't match, we want to hear about it.

Votes feed a standard Glicko-2 implementation — the rating system used in chess and competitive gaming. It tracks three values per tool: a rating, a rating deviation (RD, the uncertainty), and volatility.

• –Initial state: rating 1500, RD 350, volatility 0.06.
• –RD shrinks toward 30 as votes accumulate — more votes, more certainty.
• –Leaderboard position uses the conservative rating: rating minus 2×RD. A tool can't rank high on a lucky handful of votes.
• –Positions that fall inside each other's margin of error are merged into a shared placing — see technical draws.
• –“Both Bad” ballots don't move relative ratings but are tracked and published per tool.

Ranked positions appear in tiers: Provisional after 80+ decisive votes with RD ≤ 90, and Stable after 200+ decisive votes with RD ≤ 60 and near-complete prompt coverage. Until a tool reaches Provisional, it is listed unranked.

Every tool is evaluated under the same published benchmark protocol, voting rules, and rating math:

• –Entries follow the same benchmark protocol, voting rules, and rating math.
• –Outputs are anonymized in the arena; voters can't tell which tool they're scoring.
• –Results are published unmodified, including poor-performing outputs.
• –The methodology and output notes are available for public review.

Any AI tool that can produce 3D output usable in Blender is eligible: mesh generators, assistants, agents, and addons. To request inclusion (or removal, or a correction to a profile), email hello@blenderai.org — inclusion is free and cannot be paid for.

• –Blind voting measures perceived output quality — not workflow speed, pricing, topology quality, or editability. Read the tool profiles for those dimensions.
• –Assistants build geometry procedurally while generators produce sculpted meshes; some prompts inherently favor one approach. The prompt set is designed to balance this, and per-category results are published.
• –Tools update frequently. Each season pins tool versions where possible and re-benchmarks from scratch.
• –Competitors returning to a new season with their previous artifacts are being scored on older work than entrants who re-ran the set. The generation note on each output says which it is, and the prompt set version is pinned so the comparison stays valid.