Models represented2119 new configurations + 2 Claude references
Independent graph cases63 tasks · 32 and 64 nodes
Returned model responses502 / 50424 slots per model · 2 Step API failures
Pilot cost, including uncertain charges$12.4448 matched Claude responses reused
Exploratory pilot · six graphs per model · compute varies.
Pilot settings and scope
Each model answered six graph questions in four naming conditions. We requested 8,192 completion tokens; Grok’s provider counted hidden reasoning separately, reporting up to 36,156 tokens. The sample is small and the compute differs, so use it to explore patterns rather than rank models precisely.
Loading pilot results…
Model-family results
Original and renamed questions · correct / returned responses.
Pilot results · Claude rows reuse historical responses.
Model / family
Returned / planned
Select a model for details. Accuracy uses returned responses, including invalid answers and cutoffs.
Responses across the tested families
0%25%50%75%100%
CorrectWrong answerInvalid answersToken cutoff
Bars show returned responses. API failures appear in the table.
Where the Claude results came fromClaude response sources
Matched subset: Opus 24/24. Sonnet 23/24.
These six graphs come from a larger 24-graph replication. In that full run, Opus got 91/96 answers right and had five cutoffs. Earlier development runs had three completed errors; the higher-budget holdouts had seven cutoffs. View every Opus run → View every Sonnet run →
Full benchmark
The full Hard v1 benchmark
Hard v1 tests five configurations on 120 fresh graphs, six tasks and eight conditions. Its questions and settings differ from this pilot.