Evaluation / Saved answers

Compare your
saved answers.

Upload answers to the original and renamed questions to check accuracy, agreement and how many correct answers survived.

Paired answers

Local processing

Use one record per query, with an original, neutral-renamed and semantic-renamed prediction. Files stay in this browser.

No file loaded.

Your evaluation

→

Add answers to see the comparison.

Upload your answers or try the example to see the scores here.

File format

How to format your file

Each record requires a unique id, a task, ground_truth, original, neutral and semantic. Predictions must already be parsed: yes/no, a nonnegative integer for shortest path, or unclear. Ground truth cannot be unclear.

Task identifiers: connectivity, shortest_path, triangle_detection, common_neighbor.

Download example JSON →

Which answers count?

Accuracy uses all uploaded queries. Consistency excludes pairs containing unclear. CFM includes every initially correct query; an unclear semantic answer counts as a failure. Empty denominators display —.

For the complete protocol

Provide an extracted answer for each of the three conditions. This tool does not call a model, read free-form responses or test format, order or distractors. Use the released Python auditor for native results_*.json files, bootstrap intervals and the full diagnostics.

Full evaluation workflow →