Original paper protocol

For the new 32/64-node benchmark, explore the Hard v1 leaderboard, methodology, or run your model on Hard v1.

Reported results / Camera-ready Table 2

Results from
the original paper.

Compare four configurations on accuracy and on how their answers change with format, order and node names.

Paper-reported values. Higher is better for Accuracy and CFM; lower is better for the other diagnostics.
ModelAccuracy ↑1 − FS ↓OS ↓Δsem ↓1 − DR ↓CFM ↑
Values in %. Δsem is in percentage points. Darker cells indicate worse values within a column.JSON →
Read the pattern

Two patterns in the results

A small semantic gap does not always mean stable answers. Read it alongside the other scores.

A

Answers change with descriptive names

DeepSeek's consistency falls from 96.9% to 70.1% under semantic names while format, order and distractor stability remain high. gpt-5.4-mini retains 53.3% of originally correct answers.

B

Answers change across several conditions

Qwen and Llama were less consistent across several tests. Their semantic gaps are smaller (6.1 and 9.1 points) because consistency was already low with neutral names.

Metric field guide

How to read the scores

Accuracy

The share of original answers that match the answer calculated from the graph.

1 − FS

How often answers disagree across the six pairings of four graph formats.

OS

Order sensitivity: how often answers change when the same graph is listed in a different order.

Δsem = ICN − ICM

Agreement with neutral names minus agreement with descriptive names. A larger positive gap means agreement fell more after descriptive names were introduced.

1 − DR

How often answers change when unrelated graph elements are added without changing the correct answer.

CFM

Among originally correct answers, the share that stays correct after semantic renaming.