Ciro Santilli @cirosantilli 37

Questions available to anyone under Hugging Face login / .zip with password, but you have to promise not to post them online. Lol. Either do the thing or don't.

Numerical solution:

Earliest known public leak: github.com/lucky-bai/projecteuler-solutions/issues/93

Programs:

Math almost saturated as of 2025 release, so meh:

Numerical solution:

Earliest known public leak: github.com/lucky-bai/projecteuler-solutions/issues/87

Programs:

In 2025 they announced huge improvements on ARC-AGI-2, but they only tested on the public dataset, so the potential for contamination is overwhelming.

Numerical solution:

Earliest known public leak:

Programs:

Not too exciting because of the high school knowledge olympiad level, but respectable.

This section is about formalization efforts of specific fields of mathematics.

Numerical solution:

Earliest known public leak:

Programs:

Numerical solution:

Earliest known public leak: x.com/cirosantilli/status/1990363555309490585

Programs:

This one doesn't seem to exciting to be honest, but it might be useful. Sample question:and it expects the correct answer down to the cents:It should be noted that Project Euler has such "precision matters" problems.

Even more than in other areas of benchmarking, in maths where you only have a right or wrong answer, and it is costly to come up with good sample problems, some benchmarks have adopted private test data sets.

The situation is kind of sad, in that ideally we should have open data sets and only test them on models that were trained on data exclusively published before the problem publish date.

However this is not practical for the following reasons:

Perhaps the ideal scenario therefore is what ARC-AGI has done: give a sizeable public dataset, which you feel is highly representative of the difficulty level of the private test data, while at the same time holding out some private test data. Half half seems reasonable.

This way, reproducible models can actually self test themselves reliably on the open data, while the closed data can then be used for the cases where the open data can't be used.