Generate
Generate graphs and calculate the correct answers from their edges.
For the new 32/64-node benchmark, explore the Hard v1 leaderboard, methodology, or run your model on Hard v1.
Test whether a model gives the same answer when the same graph is written differently.
Change the names or format and watch the answer stay the same. Your browser solves this example directly; no model is called.
Descriptive names follow the number of connections, as in the original generator. The question uses the new names too.
| Task | Question | Observed target distribution |
|---|---|---|
| Connectivity | Does a path exist between two nodes? | 300 yes / 0 no |
| Shortest path | How many edges are in the shortest path? | Lengths 1–5: 101 / 143 / 50 / 4 / 2 |
| Triangle | Does the node belong to a triangle? | 184 yes / 116 no |
| Common neighbor | Do two nodes share a neighbor? | 202 yes / 98 no |
Connected Erdős–Rényi graphs, edge probability 0.3, deterministic seed 42. Three queries per graph-task pair. Connectivity is all-positive in this dataset.
Generate graphs and calculate the correct answers from their edges.
Change the format, order or names, or add graph elements that leave the answer unchanged.
Ask the model each version at temperature 0, then read its yes/no or integer answer.
Check whether answers agree and whether correct answers stay correct.