Earlier study / Model-family pilot

A first look at
21 models.

Earlier pilot · 2 Nov 2026

21 models · 6 graph questions · 4 naming conditions.

Models represented2119 new configurations + 2 Claude references
Independent graph cases63 tasks · 32 and 64 nodes
Returned model responses502 / 50424 slots per model · 2 Step API failures
Pilot cost, including uncertain charges$12.4448 matched Claude responses reused

Exploratory pilot · six graphs per model · compute varies.

Pilot settings and scope

Each model answered six graph questions in four naming conditions. We requested 8,192 completion tokens; Grok’s provider counted hidden reasoning separately, reporting up to 36,156 tokens. The sample is small and the compute differs, so use it to explore patterns rather than rank models precisely.

Loading pilot results…
Where the Claude results came from
Claude response sources

Matched subset: Opus 24/24. Sonnet 23/24.

These six graphs come from a larger 24-graph replication. In that full run, Opus got 91/96 answers right and had five cutoffs. Earlier development runs had three completed errors; the higher-budget holdouts had seven cutoffs. View every Opus run →
View every Sonnet run →

Full benchmark

The full Hard v1 benchmark

Hard v1 tests five configurations on 120 fresh graphs, six tasks and eight conditions. Its questions and settings differ from this pilot.

Open the Hard v1 leaderboard →