Answers change with descriptive names
DeepSeek's consistency falls from 96.9% to 70.1% under semantic names while format, order and distractor stability remain high. gpt-5.4-mini retains 53.3% of originally correct answers.
For the new 32/64-node benchmark, explore the Hard v1 leaderboard, methodology, or run your model on Hard v1.
Compare four configurations on accuracy and on how their answers change with format, order and node names.
| Model | Accuracy ↑ | 1 − FS ↓ | OS ↓ | Δsem ↓ | 1 − DR ↓ | CFM ↑ |
|---|
A small semantic gap does not always mean stable answers. Read it alongside the other scores.
DeepSeek's consistency falls from 96.9% to 70.1% under semantic names while format, order and distractor stability remain high. gpt-5.4-mini retains 53.3% of originally correct answers.
Qwen and Llama were less consistent across several tests. Their semantic gaps are smaller (6.1 and 9.1 points) because consistency was already low with neutral names.
The share of original answers that match the answer calculated from the graph.
How often answers disagree across the six pairings of four graph formats.
Order sensitivity: how often answers change when the same graph is listed in a different order.
Agreement with neutral names minus agreement with descriptive names. A larger positive gap means agreement fell more after descriptive names were introduced.
How often answers change when unrelated graph elements are added without changing the correct answer.
Among originally correct answers, the share that stays correct after semantic renaming.