MisGraph / Hard v1

Graph reasoning benchmark

Correct answers did not always stay correct after renaming. 53.3%–81.5% of initially correct answers stayed correct after degree-reversed renaming, depending on the configuration.

120 graphs6 tasks8 conditions16,384 token cap5 evaluated configurations4,800 responsesProtocol ↗

Hard v1 · results snapshot 2026-11-03 · 16,384-token cap; costs include recovery attempts.

New: Claude’s triangle accuracy fell from 86.5% to 62.5% after renaming → 96 fresh graphs · zero cutoffs. Separate from Hard v1.

Leaderboard

Download CSV ↓

5 evaluated configurations · full release · original accuracy, highest first

Scroll sideways to see all scores →

Renaming survival: originally correct → still correct with degree-reversed names.
Model / providerOriginal accuracyRenaming survivalCutoffsCoverageCost / 1,000
DeepSeek V4.1 Flashsail-research/fp4 · FP473.3%n=12073.9%n=88 initially correct answers32.1%308/960960/960$1.61
MiniMax M3.1 Flashtenstorrent/fp8 · FP867.5%n=12081.5%n=81 initially correct answers21.5%206/960960/960$1.58
GLM-5.3baidu/fp8 · FP863.3%n=12080.3%n=76 initially correct answers38.8%372/960960/960$9.42
Ling 3.1 Flashnovita · UNKNOWN39.2%n=12076.6%n=47 initially correct answers62.2%597/960960/960$3.07
Granite 4.2 8Bcoreweave/bf16 · BF1612.5%n=12053.3%n=15 initially correct answers89.8%862/960960/960$2.49

These are the full Hard v1 results. Enable JavaScript to use the filters and charts. Metric definitions and intervals · All scores (JSON)

Development study16K vs. 32K token study ↗Four configurations · 24 graphs · two token limitsResearch archiveEarlier runs and responses ↗Paper study, family pilot and development runsReproducibilityScores and confidence intervals ↓Download the published scores and confidence intervals
What changes when you rename a node?

Only the labels change in this illustration. The edges and correct answer stay the same.

Try renaming the nodes
Neutral names. Six nodes, eight edges.
Release identifiers and settings

Protocol hard-v1-16k-8db578dd652c
SHA-256 8db578dd652caa37fcbe7ac745bfde5f18b111dc64d5156c209949c63315b4d7

We use high reasoning effort where supported, default sampling and no tools. Each model uses a fixed provider. Reasoning settings and token counts work differently across providers.

Download settings and file hashes
Cite this benchmark release

Hard v1 · results snapshot . Use this citation when referring to these results.

@misc{hu2026misgraphhardv1,
  author = {Hu, Lifan},
  title = {{MisGraph} Hard v1: Graph Reasoning under Node Renaming},
  year = {2026},
  howpublished = {\url{https://anormalm.com/misgraph/}},
  note = {Hard v1, 16,384-token protocol. Results snapshot 2026-11-03. Protocol SHA-256: 8db578dd652caa37fcbe7ac745bfde5f18b111dc64d5156c209949c63315b4d7}
}