MisGraph Hard v1 / Methodology

Methodology

We ask each model a graph question, change how the graph is presented, and check whether its answer stays correct.

01 / Generate120 independent graphs6 tasks × 2 sizes × 10 seeds
02 / Transform8 prompt conditionsThe correct answer stays the same
03 / Query960 requests per modelFrozen prompts · no tools · 16,384-token cap
04 / AuditAccuracy and consistencyVerified answers · paired comparisons · confidence intervals

Six graph tasks

TaskTarget answer
Weighted shortest pathFind the path with the lowest total edge weight between two nodes.
Shortest-path countCount how many shortest paths connect two nodes.
Exact-walk countCount walks of a given length. A walk may revisit nodes and edges.
Triangle countCount the triangles in the graph, counting each one once.
Minimum cutFind the lowest total edge capacity that must be removed to disconnect two nodes.
Constrained reachabilityCheck whether two nodes are still connected after specified nodes are removed: 1 for yes, 0 for no.

Each task uses graphs with 32 or 64 nodes, generated from ten fixed seeds per size. Half of the reachability answers are yes and half are no.

Eight prompt variants

Original → identical repeat

Ask the same question again to see how often the model changes its answer on its own.

Neutral → aligned → reversed names

Use neutral names, names that follow the number of connections, or names that reverse that ranking. For example, a highly connected node can be called Leaf.

JSON format → shuffled order

Write the same graph as JSON, or shuffle the order in which its nodes and edges appear.

Disconnected addition

Add a separate three-node path that does not change the answer.

Metric definitions
  • Accuracy: The share of returned responses that are correct. Original accuracy uses one question per graph. All-condition accuracy includes all eight versions of each question; these are related observations.
  • Agreement: How often the original and changed prompts produce the same answer. We count only pairs where both answers can be read, and show the number of pairs.
  • Renaming survival: Of the questions answered correctly at first, the share still correct after degree-reversed renaming. Both responses must be available. Invalid answers and unsuccessful cutoffs count as failures.
  • Cutoffs: Responses that hit the token limit without a correct final answer. We report completed wrong answers and invalid answers separately.
  • Matched semantic gap: Agreement with neutral names minus agreement with reversed names, using questions where all three answers can be read. This metric is included in the download.

Model details include 95% Wilson confidence intervals based on graphs. The download also includes 2,000 bootstrap draws, stratified by task and size. A score is undefined when there are no eligible observations.

How we ran the models

We tested five model and provider configurations with a 16,384-token completion limit. We requested high reasoning effort where available, enabled reasoning otherwise, and used default sampling without tools. The download includes provider details, precision, original responses, token usage, costs and request logs.

Ling ran on Novita, which did not disclose its precision. Its original provider rejected requests without charging or returning model output. The other configurations use DeepSeek FP4, GLM FP8, MiniMax FP8 and Granite BF16. These scores describe the tested configurations; providers differ in how they count tokens and implement reasoning.

We retried infrastructure failures only. We kept every wrong, invalid or truncated model answer. Eight requests had an approved third attempt after two provider failures. All 4,800 planned responses were collected. The total is $17.44515, including a $0.30615 upper estimate for unconfirmed charges.

Scope and reproducibility

Hard v1 covers six tasks on synthetic graphs at two sizes. Descriptive names follow node degree, not a node’s role in the task. Renaming also changes tokenization. The identical-repeat condition shows how often answers change without any edit to the prompt.

All 120 answers were independently checked. The release includes the answer certificates, fixed prompts and code, file hashes, complete attempt logs and final audit. The paper study and 21-model pilot use different datasets, so their scores are shown separately.