Ciro Santilli @cirosantilli 37

Even more than in other areas of benchmarking, in maths where you only have a right or wrong answer, and it is costly to come up with good sample problems, some benchmarks have adopted private test data sets.

The situation is kind of sad, in that ideally we should have open data sets and only test them on models that were trained on data exclusively published before the problem publish date.

However this is not practical for the following reasons:

Perhaps the ideal scenario therefore is what ARC-AGI has done: give a sizeable public dataset, which you feel is highly representative of the difficulty level of the private test data, while at the same time holding out some private test data. Half half seems reasonable.

This way, reproducible models can actually self test themselves reliably on the open data, while the closed data can then be used for the cases where the open data can't be used.

www.giotto.ai/Their website doesn't clearly explain their technology as of 2025.

They claim to have done some work on ARC-AGI which is cool, but no clear references to what they did or if there's anything public about it.

ndea.com/Cool. Founders are also very interested in ARC-AGI.

This section is about unofficial ARC-AGI-like problem sets.

These are interesting from both a:

github.com/neoneye/arc-dataset-collection contains a fantastic collection of such datasets, with visualization at: neoneye.github.io/arc/

I've created a quick fork of ARC-DSL which defines a hand crafted Domain Specific Language (DSL) approach to help solve ARC-AGI problems.

I basically just merged outstanding pull requests on the original repo that were needed to make things run.

It would be cool to see if those rules also solve ARC-AGI-2 problems well, but lazy now.

ARC-AGI-2 is a very interesting benchmark which mixes some symbolic and other visual elements, and is readily solvable by non-expert humans, but has so far resisted transformers to a large degree.

Part of me would like to focus more on less visual aspects of AI, but it is still of interest.

It is funny how many early (semi)-retired fintech/bigtech bros that are interested in the project, I saw several of them on the forums.

I'd be tempted if I were in that position too I must confess. Maybe in 15 years time for me the way things are looking.

Kudos to these people who do something cool and open when they don't need money: www.reddit.com/r/Fire/comments/15x4w7r/comment/jx7dn16/ It is also the case of Jimmy Wales from Wikipedia for example, who used to work in finance.

Announcements: