What happens as graphs get harder?
GPT-6.1 Sol answered 285 of 288 prompts correctly. Other configurations made more errors or ran out of tokens.
Collection finished 13 Nov 2026 · reviewed 16 Nov 2026 · separate from Hard v1.
Results by naming condition
Correct / returned responses. Each condition has 72 planned prompts per model.
| Configuration | Original | Identical repeat | Neutral names | Degree-reversed names | Cutoffs / responses | Coverage |
|---|---|---|---|---|---|---|
| GLM 5.3No responses | — | — | — | — | — | 0/288 |
| MiniMax M3.1 FlashFull coverage | 49/72 | 54/72 | 51/72 | 41/72 | 46/288 | 288/288 |
| Ling 3.1 FlashFull coverage | 37/72 | 41/72 | 36/72 | 32/72 | 140/288 | 288/288 |
| Granite 4.2 8BPartial coverage | 17/72 | 14/72 | 14/71 | 10/72 | 232/287 | 287/288 |
| GPT-6.1 SolFull coverage | 72/72 | 72/72 | 70/72 | 71/72 | 1/288 | 288/288 |
| Claude Sonnet 5.5Full coverage | 60/72 | 65/72 | 57/72 | 58/72 | 16/288 | 288/288 |
| Gemini 3.8 FlashPartial coverage | 31/33 | 35/39 | 29/35 | 32/36 | 10/143 | 143/288 |
| Qwen3.8 27BPartial coverage | 45/71 | 41/72 | 43/72 | 38/72 | 103/287 | 287/288 |
| Llama 4 MaverickFull coverage | 4/72 | 4/72 | 5/72 | 2/72 | 0/288 | 288/288 |
GLM returned no scored responses. Gemini has 143/288; Granite and Qwen each lack one. Missing responses are never scored as zero.
How to read these scores
Renaming preserves the graph and its correct answer. An identical-prompt repeat measures ordinary answer variation. Wrong answers, invalid answers and cutoffs remain in the returned-response denominator. A shared token cap does not give every model equal computation.
The paired analysis compares renamed accuracy with identical-repeat accuracy. None of the 18 planned contrasts passes Holm correction. Confidence intervals and all denominators are in the analysis JSON. There are four independent graphs per task and size; the four conditions are not independent samples.
A closer look: Claude counting triangles
On four 32-node graphs, Sonnet 5.5 was correct with original labels, identical repeats and degree-reversed labels. With neutral renaming, it missed all four.
| Case | Exact count | Claude, neutral names | Error |
|---|---|---|---|
| Graph 1 | 201 | 198 | -3 |
| Graph 2 | 203 | 205 | +2 |
| Graph 3 | 202 | 201 | -1 |
| Graph 4 | 216 | 215 | -1 |
This case study was selected after looking at the results. It identifies a pattern to test on fresh graphs, not an established cause. Check the case audit.
Follow-up complete: see the predeclared test on 96 fresh graphs →
Protocol, coverage and downloads
Six tasks: weighted shortest path, shortest-path count, exact-walk count, triangle count, minimum cut and constrained reachability. Each uses four graphs at each of three sizes. Providers and settings were fixed per configuration. Collection stopped when no eligible work remained; there are no outstanding reservations.
GLM was blocked by provider failures; Gemini stopped after a protocol-invalid response. Granite and Qwen each exhausted the permitted infrastructure recovery on one slot. Those gaps limit comparisons.
All 72 exact-answer certificates and 288 prompt checks passed. The study is exploratory development work; these scores do not replace the frozen leaderboard.
Frozen plan SHA-256b2385971849cf0c79362d4852492230c2ed058566726adb0059ab75df53dbd3c