Same triangles.
Different names. More errors.
Claude Sonnet 5.5’s accuracy fell from 86.5% to 62.5% when we renamed the nodes in 96 fresh graphs.
One model configuration · one graph task · separate from Hard v1.
The planned comparison
Triangle-counting accuracy
Same graphs, edges and correct answers. Only the node labels change.
| Condition | Correct / graphs | Accuracy |
|---|---|---|
| Original labels | 83 / 96 | 86.5% |
| Renamed labels | 60 / 96 | 62.5% |
24.0 points lower
95% interval: 11.5–36.5 points lower · paired test p = 0.00043
Renaming turned 32 correct answers into wrong ones, and 9 wrong answers into correct ones. Every response finished without a token cutoff.
Does repeating the prompt explain it?
We asked each original question three times and tested three neutral renamings. Original accuracy stayed near 86%; renamed accuracy averaged 67%.
See all three repetitions
| Repetition | Original | Renamed |
|---|---|---|
| 1 | 83 / 96 | 60 / 96 |
| 2 | 83 / 96 | 63 / 96 |
| 3 | 82 / 96 | 69 / 96 |
Across all repetitions: original 248/288 (86.1%), renamed 192/288 (66.7%). The average within-graph drop was 19.4 points (95% interval: 12.5–26.4). This secondary analysis is exploratory; it resamples the 96 graphs, not 576 independent observations.
Answers differed in 71/288 pairs of unchanged repetitions and 159/288 pairs of renamings. These within-graph pairs are correlated and are reported descriptively.
What this tells us
The renaming effect held up on fresh graphs in a comparison fixed before collection. Running out of tokens does not explain these errors.
This result covers Claude Sonnet 5.5 on this family of 32-node graphs. It does not identify the model’s internal failure mechanism. Read the analysis →
How we ran the study
We generated 96 fresh Watts–Strogatz graphs (32 nodes, initial degree 10, rewiring probability 0.22), requiring connectedness, at least 80 triangles and 40 edges shared by multiple triangles. None is isomorphic to a stored prior case or another study graph. Exact counts were checked by three independent methods.
Every request starts a new conversation. Each graph has three byte-identical original prompts and three neutral renamings using the same X000–X031 vocabulary. Node and edge occurrence order is preserved under each mapping. Prompt byte lengths match; this does not establish identical internal token representations.
Claude Sonnet 5.5 ran through OpenRouter’s pinned Anthropic route, with high reasoning, a 16,384-token completion cap, default sampling, no tools and no provider fallback. Graph blocks and requests within each block were randomized.
The single primary test compares original_0 with neutral_0 across all 96 graphs: exact two-sided McNemar, α = 0.05. All designated pairs are present. Confidence intervals use 20,000 graph-bootstrap draws, seed 20261116. The earlier four selected cases motivated this study but were not reused.
Frozen plan SHA-25628162bccb62a4999721187311f627e8cd644574c85991b3da4bf3d4b53b7d31f
Coverage, tokens and collection cost
All 576 responses were scored: 440 correct, 136 wrong, no invalid answers and no cutoffs. Two infrastructure failures received one recovery attempt each, for 578 total attempts. No returned answer was retried.
Successful responses used 890,496 prompt tokens and 2,354,275 completion tokens, including 2,339,210 reasoning tokens. Failed attempts without reported usage are excluded from these token totals.
Provider-reported cost: $25.323742. Uncertain charges retained: $0.355096. Total accounted cost: $25.678838, with no outstanding reservations. Graph certificates, serialized prompts, frozen hashes, retry history and the local archive were independently verified.