Original paper protocol

For the new 32/64-node benchmark, explore the Hard v1 leaderboard, methodology, or run your model on Hard v1.

Reproducibility / Open evaluation

Run the protocol.
Check the results.

Use the Python tools to generate questions, query a model and check its responses.

01 / Environment

Install the tools

The released code supports Python 3.10+. Start with its deterministic protocol tests.

git clone https://github.com/Anormalm/graph_evidence.git
cd graph_evidence/code
python -m venv .venv
# macOS / Linux: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python -m unittest discover -s tests -v
02 / Model inference

Query your model

Set your provider key through the appropriate environment variable, such as OPENROUTER_API_KEY, on your own machine. You can also use the browser runner with your own key and a compatible endpoint.

python run_api_models.py --model deepseek/deepseek-v4-flash --endpoint openrouter --output-dir ../results

Use the runner's --help for available endpoints. Temperature is 0; reasoning settings vary by model. Preserve the model ID, configuration and raw generations with each run.

03 / Offline audit

Check the answers

python audit_results.py --results-dir ../results --output ../results/audit_summary.json

The auditor reads saved responses and reports scores, confidence intervals, parsing rates and possible renaming failures. This step makes no model calls.

Try the paired-answer evaluator →
Data provenance

What you
can reproduce.

Source: released repository and camera-ready paper

The historical raw generations behind Table 2 were absent from the source checkout. The original CFM implementation was also absent; the released auditor adds that metric from the paper's definition.

The corrected code fixes query remapping, unclear-answer handling, the denominator for order sensitivity and checks on distractors. Label new results as corrected-protocol runs: they are not exact reproductions of the published table.

Scope and interpretation

The paper uses small synthetic graphs in four formats. Every connectivity answer is yes. Descriptive names follow decreasing node degree, so they do not test deliberately misleading task roles. The shared prompt also does not say that node names are arbitrary. Keep these choices in mind when interpreting the results.

Read the full provenance record →
Paper & citation

Graph Evidence Is Not Text Evidence

Serialization Faithfulness as Structural Invariance in Graph-Grounded Language Models.
Lifan Hu · National University of Singapore | GroundLM Workshop, EMNLP 2026.

@misc{hu2026graphevidence,
  title = {Graph Evidence Is Not Text Evidence: Serialization Faithfulness as Structural Invariance in Graph-Grounded Language Models},
  author = {Hu, Lifan},
  year = {2026},
  note = {GroundLM Workshop, EMNLP 2026},
  url = {https://github.com/Anormalm/graph_evidence}
}