Original paper protocol

For the new 32/64-node benchmark, explore the Hard v1 leaderboard, methodology, or run your model on Hard v1.

Benchmark / Task suite

Explore the
paper’s graph tasks.

Test whether a model gives the same answer when the same graph is written differently.

100connected graphs
4reasoning tasks
1,200query instances
5–15nodes per graph
Interactive / Structure explorer

Try renaming
the nodes.

Change the names or format and watch the answer stay the same. Your browser solves this example directly; no model is called.

Neutral names
Correct answer
Prompt sent to a model

Descriptive names follow the number of connections, as in the original generator. The question uses the new names too.

Task suite

Four questions about structure.

TaskQuestionObserved target distribution
ConnectivityDoes a path exist between two nodes?300 yes / 0 no
Shortest pathHow many edges are in the shortest path?Lengths 1–5: 101 / 143 / 50 / 4 / 2
TriangleDoes the node belong to a triangle?184 yes / 116 no
Common neighborDo two nodes share a neighbor?202 yes / 98 no

Connected Erdős–Rényi graphs, edge probability 0.3, deterministic seed 42. Three queries per graph-task pair. Connectivity is all-positive in this dataset.

Evaluation protocol

How the
comparison works

01

Generate

Generate graphs and calculate the correct answers from their edges.

02

Transform

Change the format, order or names, or add graph elements that leave the answer unchanged.

03

Query

Ask the model each version at temperature 0, then read its yes/no or integer answer.

04

Compare

Check whether answers agree and whether correct answers stay correct.

Run the released protocol →