Which answers count?
Accuracy uses all uploaded queries. Consistency excludes pairs containing unclear. CFM includes every initially correct query; an unclear semantic answer counts as a failure. Empty denominators display —.
Upload answers to the original and renamed questions to check accuracy, agreement and how many correct answers survived.
Use one record per query, with an original, neutral-renamed and semantic-renamed prediction. Files stay in this browser.
No file loaded.
Upload your answers or try the example to see the scores here.
Each record requires a unique id, a task, ground_truth, original, neutral and semantic. Predictions must already be parsed: yes/no, a nonnegative integer for shortest path, or unclear. Ground truth cannot be unclear.
Task identifiers: connectivity, shortest_path, triangle_detection, common_neighbor.
Accuracy uses all uploaded queries. Consistency excludes pairs containing unclear. CFM includes every initially correct query; an unclear semantic answer counts as a failure. Empty denominators display —.
Provide an extracted answer for each of the three conditions. This tool does not call a model, read free-form responses or test format, order or distractors. Use the released Python auditor for native results_*.json files, bootstrap intervals and the full diagnostics.