Original paper protocol

For the new 32/64-node benchmark, explore the Hard v1 leaderboard, methodology, or run your model on Hard v1.

Evaluation / Bring your own service

Test your model
on the paper’s tasks.

Connect your API and choose a test to see how your model handles different ways of writing the same graph.

01 Connect02 Choose a scope03 Run & compare

Connect your model

Connect your API

Use a chat-completions-compatible API or local server. Requests go directly from this browser to the endpoint below.

The site does not save your key or send it to a separate benchmark server. Only enter a key for an endpoint you trust.

Choose your tests

Smoke and pilot cover one graph at each size: 5, 8, 10, 12 and 15 nodes.

Tasks
Diagnostics
Request settings
Before you run
Loading protocol...

Your provider charges for accepted requests. No automatic retries. Stopping cancels browser requests; your service may finish requests it already accepted.

Know what you measured

What this
run measures.

The questions come from the corrected Python protocol. Repeated questions have separate IDs so no response is overwritten. Your run produces new results; it does not recreate the paper’s original responses.

How scoring works

Accuracy uses baseline questions that received a response. Consistency compares pairs where both answers can be read. CFM is the share of initially correct answers that remain correct under semantic renaming; unclear semantic answers count as failures.

Each score shows how many answers it uses. Partial runs use the available comparisons, with API failures listed separately. Use the Python auditor to add bootstrap confidence intervals.

Service compatibility

The endpoint must accept a chat-completions request and return text in choices[0].message.content. Native Messages and Responses APIs require a compatible gateway. The service must allow this site's browser origin through CORS.

For local services, serve this website locally too. A remote HTTPS page may be blocked from calling an HTTP local server. Browser request behavior.