Triangle confirmation v1 / Reviewed 17 Nov 2026

Same triangles.
Different names. More errors.

Claude Sonnet 5.5’s accuracy fell from 86.5% to 62.5% when we renamed the nodes in 96 fresh graphs.

96 graphs32 nodes each576 / 576 responses0 cutoffs

One model configuration · one graph task · separate from Hard v1.

The planned comparison

Triangle-counting accuracy

Original labels86.5%
Renamed labels62.5%

Same graphs, edges and correct answers. Only the node labels change.

First designated original / renamed pair on each of 96 independent graphs.
ConditionCorrect / graphsAccuracy
Original labels83 / 9686.5%
Renamed labels60 / 9662.5%

24.0 points lower
95% interval: 11.5–36.5 points lower · paired test p = 0.00043

Renaming turned 32 correct answers into wrong ones, and 9 wrong answers into correct ones. Every response finished without a token cutoff.

Does repeating the prompt explain it?

We asked each original question three times and tested three neutral renamings. Original accuracy stayed near 86%; renamed accuracy averaged 67%.

See all three repetitions
Correct / 96 graphs. The original prompts are identical; the renamed prompts use three different mappings.
RepetitionOriginalRenamed
183 / 9660 / 96
283 / 9663 / 96
382 / 9669 / 96

Across all repetitions: original 248/288 (86.1%), renamed 192/288 (66.7%). The average within-graph drop was 19.4 points (95% interval: 12.5–26.4). This secondary analysis is exploratory; it resamples the 96 graphs, not 576 independent observations.

Answers differed in 71/288 pairs of unchanged repetitions and 159/288 pairs of renamings. These within-graph pairs are correlated and are reported descriptively.

What this tells us

The renaming effect held up on fresh graphs in a comparison fixed before collection. Running out of tokens does not explain these errors.

This result covers Claude Sonnet 5.5 on this family of 32-node graphs. It does not identify the model’s internal failure mechanism. Read the analysis →

How we ran the study

We generated 96 fresh Watts–Strogatz graphs (32 nodes, initial degree 10, rewiring probability 0.22), requiring connectedness, at least 80 triangles and 40 edges shared by multiple triangles. None is isomorphic to a stored prior case or another study graph. Exact counts were checked by three independent methods.

Every request starts a new conversation. Each graph has three byte-identical original prompts and three neutral renamings using the same X000–X031 vocabulary. Node and edge occurrence order is preserved under each mapping. Prompt byte lengths match; this does not establish identical internal token representations.

Claude Sonnet 5.5 ran through OpenRouter’s pinned Anthropic route, with high reasoning, a 16,384-token completion cap, default sampling, no tools and no provider fallback. Graph blocks and requests within each block were randomized.

The single primary test compares original_0 with neutral_0 across all 96 graphs: exact two-sided McNemar, α = 0.05. All designated pairs are present. Confidence intervals use 20,000 graph-bootstrap draws, seed 20261116. The earlier four selected cases motivated this study but were not reused.

Frozen plan SHA-256
28162bccb62a4999721187311f627e8cd644574c85991b3da4bf3d4b53b7d31f

Coverage, tokens and collection cost

All 576 responses were scored: 440 correct, 136 wrong, no invalid answers and no cutoffs. Two infrastructure failures received one recovery attempt each, for 578 total attempts. No returned answer was retried.

Successful responses used 890,496 prompt tokens and 2,354,275 completion tokens, including 2,339,210 reasoning tokens. Failed attempts without reported usage are excluded from these token totals.

Provider-reported cost: $25.323742. Uncertain charges retained: $0.355096. Total accounted cost: $25.678838, with no outstanding reservations. Graph certificates, serialized prompts, frozen hashes, retry history and the local archive were independently verified.