Development study / Difficulty v1

What happens as graphs get harder?

GPT-6.1 Sol answered 285 of 288 prompts correctly. Other configurations made more errors or ran out of tokens.

72 graphs6 tasks16 / 24 / 32 nodes16,384 token cap

Collection finished 13 Nov 2026 · reviewed 16 Nov 2026 · separate from Hard v1.

Scored responses2,157 / 2,592435 missing or protocol-invalid
Full coverage5 / 9configurations
Accounted cost$27.00$26.80 reported + $0.20 uncertain

Results by naming condition

Correct / returned responses. Each condition has 72 planned prompts per model.

Fixed protocol order; partial rows are descriptive, not a ranking. Scroll horizontally on small screens.
ConfigurationOriginalIdentical repeatNeutral namesDegree-reversed namesCutoffs / responsesCoverage
GLM 5.3No responses—————0/288
MiniMax M3.1 FlashFull coverage49/7254/7251/7241/7246/288288/288
Ling 3.1 FlashFull coverage37/7241/7236/7232/72140/288288/288
Granite 4.2 8BPartial coverage17/7214/7214/7110/72232/287287/288
GPT-6.1 SolFull coverage72/7272/7270/7271/721/288288/288
Claude Sonnet 5.5Full coverage60/7265/7257/7258/7216/288288/288
Gemini 3.8 FlashPartial coverage31/3335/3929/3532/3610/143143/288
Qwen3.8 27BPartial coverage45/7141/7243/7238/72103/287287/288
Llama 4 MaverickFull coverage4/724/725/722/720/288288/288

GLM returned no scored responses. Gemini has 143/288; Granite and Qwen each lack one. Missing responses are never scored as zero.

How to read these scores

Renaming preserves the graph and its correct answer. An identical-prompt repeat measures ordinary answer variation. Wrong answers, invalid answers and cutoffs remain in the returned-response denominator. A shared token cap does not give every model equal computation.

The paired analysis compares renamed accuracy with identical-repeat accuracy. None of the 18 planned contrasts passes Holm correction. Confidence intervals and all denominators are in the analysis JSON. There are four independent graphs per task and size; the four conditions are not independent samples.

A closer look: Claude counting triangles

On four 32-node graphs, Sonnet 5.5 was correct with original labels, identical repeats and degree-reversed labels. With neutral renaming, it missed all four.

All 16 answers finished normally. Each graph’s triangle count was checked independently.
CaseExact countClaude, neutral namesError
Graph 1201198-3
Graph 2203205+2
Graph 3202201-1
Graph 4216215-1

This case study was selected after looking at the results. It identifies a pattern to test on fresh graphs, not an established cause. Check the case audit.

Follow-up complete: see the predeclared test on 96 fresh graphs →

Why might a renamed graph get a different answer? →

Protocol, coverage and downloads

Six tasks: weighted shortest path, shortest-path count, exact-walk count, triangle count, minimum cut and constrained reachability. Each uses four graphs at each of three sizes. Providers and settings were fixed per configuration. Collection stopped when no eligible work remained; there are no outstanding reservations.

GLM was blocked by provider failures; Gemini stopped after a protocol-invalid response. Granite and Qwen each exhausted the permitted infrastructure recovery on one slot. Those gaps limit comparisons.

All 72 exact-answer certificates and 288 prompt checks passed. The study is exploratory development work; these scores do not replace the frozen leaderboard.

Frozen plan SHA-256
b2385971849cf0c79362d4852492230c2ed058566726adb0059ab75df53dbd3c