Original → identical repeat
Ask the same question again to see how often the model changes its answer on its own.
We ask each model a graph question, change how the graph is presented, and check whether its answer stays correct.
| Task | Target answer |
|---|---|
| Weighted shortest path | Find the path with the lowest total edge weight between two nodes. |
| Shortest-path count | Count how many shortest paths connect two nodes. |
| Exact-walk count | Count walks of a given length. A walk may revisit nodes and edges. |
| Triangle count | Count the triangles in the graph, counting each one once. |
| Minimum cut | Find the lowest total edge capacity that must be removed to disconnect two nodes. |
| Constrained reachability | Check whether two nodes are still connected after specified nodes are removed: 1 for yes, 0 for no. |
Each task uses graphs with 32 or 64 nodes, generated from ten fixed seeds per size. Half of the reachability answers are yes and half are no.
Ask the same question again to see how often the model changes its answer on its own.
Use neutral names, names that follow the number of connections, or names that reverse that ranking. For example, a highly connected node can be called Leaf.
Write the same graph as JSON, or shuffle the order in which its nodes and edges appear.
Add a separate three-node path that does not change the answer.
Model details include 95% Wilson confidence intervals based on graphs. The download also includes 2,000 bootstrap draws, stratified by task and size. A score is undefined when there are no eligible observations.
We tested five model and provider configurations with a 16,384-token completion limit. We requested high reasoning effort where available, enabled reasoning otherwise, and used default sampling without tools. The download includes provider details, precision, original responses, token usage, costs and request logs.
Ling ran on Novita, which did not disclose its precision. Its original provider rejected requests without charging or returning model output. The other configurations use DeepSeek FP4, GLM FP8, MiniMax FP8 and Granite BF16. These scores describe the tested configurations; providers differ in how they count tokens and implement reasoning.
We retried infrastructure failures only. We kept every wrong, invalid or truncated model answer. Eight requests had an approved third attempt after two provider failures. All 4,800 planned responses were collected. The total is $17.44515, including a $0.30615 upper estimate for unconfirmed charges.
Hard v1 covers six tasks on synthetic graphs at two sizes. Descriptive names follow node degree, not a node’s role in the task. Renaming also changes tokenization. The identical-repeat condition shows how often answers change without any edit to the prompt.
All 120 answers were independently checked. The release includes the answer certificates, fixed prompts and code, file hashes, complete attempt logs and final audit. The paper study and 21-model pilot use different datasets, so their scores are shown separately.