the previous few weeks, we’ve got seen the discharge of highly effective LLMs resembling Qwen 3 MoE, Kimi K2, and Grok 4. We’ll proceed seeing such fast enhancements within the foreseeable future, and to check the LLMs in opposition to one another, we require benchmarks. On this article, I focus on the newly launched ARC AGI 3 benchmark and why frontier LLMs battle to finish any duties on the benchmark.
Motivation
My motivation for writing this text is to remain on high of the newest developments in LLM expertise. Solely within the final couple of weeks have we seen the Kimi K2 mannequin (finest open-source mannequin when launched), Qwen 3 235B-A22B (at present finest open-source mannequin), Grok 4, and so forth. There may be a lot taking place within the LLM house, and one approach to sustain is to trace the benchmarks.
I believe the ARC AGI benchmark is especially attention-grabbing, primarily as a result of I wish to see if LLMs can match human-level intelligence. ARC AGI puzzles are made in order that people are capable of full them, however LLMs will battle.
You too can learn my article on Using Context Engineering to Considerably Improve LLM Efficiency and take a look at my web site, which accommodates all my data and articles.
Desk of Contents
Introduction to ARC AGI
ARC AGI is basically a puzzle recreation of sample matching.
- ARC AGI 1: You’re given a sequence of input-output pairs, and have to finish the sample
- ARC AGI 2: Much like the primary benchmark, performing sample matching on enter and output examples
- ARC AGI 3: Right here you’re taking part in a recreation, the place it’s a must to transfer your block into the aim space, however some required steps in between
I believe it’s cool to check out these puzzle video games and full them myself. Then, you may see LLMs initially battle with the benchmarks, after which enhance their efficiency with higher fashions. OpenAI, for instance, scored:
- 7.8% with o1 mini
- 75% with o3-low
- 88% with o3-high
As it’s also possible to see within the picture under:
Taking part in the ARC AGI benchmark
You too can attempt the ARC AGI benchmarks your self or construct an AI to carry out the duties. Go to the ARC AGI 3 web site and begin taking part in the sport.
The entire level of the video games is that you haven’t any directions, and it’s a must to determine the foundations your self. I get pleasure from this idea, because it represents determining a wholly new drawback, with none assist. This highlights your potential to be taught new environments, adapt to them, and clear up issues.
You may see a recording of me taking part in ARC AGI 3 right here, encountering the issues for the primary time. I used to be sadly unable to embed the hyperlink within the article. Nonetheless, it was tremendous attention-grabbing to check out the benchmark and picture the problem an LLM has to undergo to resolve it. I first observe the atmosphere, and what occurs once I carry out the totally different actions. An motion on this case is urgent one of many related buttons. Some actions do nothing, whereas others have an effect on the atmosphere. I then proceed to uncover the aim of the puzzle (for instance, get the item to the aim space) and attempt to obtain this aim.
Why frontier fashions obtain 0%
This text states that when frontier fashions have been examined on the ARC AGI 3 preview, they achieved 0%. This would possibly sound disappointing to some individuals, contemplating you have been in all probability capable of efficiently full a variety of the duties your self, comparatively shortly.
As I beforehand mentioned, a number of OpenAI fashions have had success with the sooner ARC AGI benchmarks, with their finest mannequin attaining 88% on the primary model. Nonetheless, initially, fashions achieved 0%, or within the low single-digit percentages.
I’ve a couple of theories for why frontier fashions weren’t capable of carry out duties on ARC AGI 3:
Context size
When engaged on ARC AGI 3, you don’t get any details about the sport. The mannequin thus has to check out quite a lot of actions, see the output of these actions (for instance, nothing occurs, or a block strikes, and so forth). The mannequin then has to guage the actions it took, together with the output, and contemplate its subsequent strikes.
I consider the motion house on ARC AGI 3 could be very giant, and it’s thus tough for the fashions to each experiment sufficient to seek out the proper motion and keep away from repeating unsuccessful actions. The fashions basically have an issue with their context size and using the complete size of it.
I lately learn an attention-grabbing article from Manus about how they develop their brokers and handle their reminiscence. You need to use methods resembling summarizing earlier context or utilizing a file system to retailer vital context. I consider this shall be key to growing efficiency on the ARC AGI 3 benchmark.
Coaching dataset
One other major cause frontier fashions are unable to finish ARC AGI 3 duties efficiently is that the duties are very totally different from their coaching dataset. LLMs will virtually at all times carry out method higher on a job if such a job (or the same one) is included within the coaching dataset. On this occasion, I consider LLMs have little coaching knowledge on working with video games, for instance. Moreover, an vital level right here can be the agentic coaching knowledge for the LLMs.
With agentic coaching knowledge, I imply knowledge the place the LLM is using instruments and performing actions. I consider we’re seeing a fast enhance in LLMs used as brokers, and thus, the proportional quantity of coaching knowledge for agentic conduct is quickly growing. Nonetheless, it could be that present frontier fashions nonetheless should not nearly as good at performing such actions, although it is going to seemingly enhance quickly within the coming months.
Some individuals will spotlight how this proves LLMs should not have actual intelligence: The entire level of intelligence (and the ARC AGI benchmark) is to have the ability to perceive duties with none clues, solely by inspecting the atmosphere. To some extent, I agree with this level, and I hope to see fashions carry out higher on ARC AGI due to elevated mannequin intelligence, and never due to benchmark chasing, an idea I discover later on this article.
Benchmark efficiency sooner or later
Sooner or later, I consider we’ll see huge enhancements in mannequin efficiency on ARC AGI 3. Principally as a result of I believe you may create AI brokers which are fine-tuned for agentic efficiency, and that optimally make the most of their reminiscence. I consider comparatively low cost enhancements can be utilized to vastly enhance efficiency, although I additionally count on costlier enhancements (for instance, the discharge of GPT-5) will carry out effectively on this benchmark.
Benchmark chasing
I believe it’s vital to depart a bit about benchmark chasing. Benchmark chasing is the idea of LLM suppliers chasing optimum scores on benchmarks, moderately than merely creating one of the best or most clever LLMs. This can be a drawback as a result of the correlation between benchmark efficiency and LLM intelligence isn’t 100%.
Within the reinforcement studying world, benchmark chasing can be known as reward hacking. A situation the place the agent figures out a approach to hack the atmosphere they’re in to attain a reward, with out correctly performing a job.
The rationale LLM suppliers do that is that at any time when a brand new mannequin is launched, individuals normally take a look at two issues:
- Benchmark efficiency
- Vibe
Benchmark efficiency is normally measured on recognized benchmarks, resembling SWE-bench and ARC AGI. Vibe testing can be a method LLMs are sometimes measured by the general public (I’m not saying it’s a great way of testing the mannequin, I’m merely saying it occurs in observe). The issue with this, nevertheless, is that I consider it’s fairly easy to impress individuals with the vibe of a mannequin, as a result of vibe checking tries some very small proportion of the motion house for the LLM. You might solely be asking it sure questions which can be found on the net, or asking it to program an utility which the mannequin has already seen 1000 situations of in its coaching knowledge.
Thus, what it’s best to do is to have a benchmark by yourself, for instance, an in-house dataset that has not been leaked to the web. Then you may benchmark which LLM works finest to your use case and prioritize utilizing this LLM.
Conclusion
On this article, I’ve mentioned LLM benchmarks and why they’re vital for evaluating LLMs. I’ve launched you to the newly launched ARC AGI 3 benchmark. This benchmark is tremendous attention-grabbing contemplating people are simply capable of full a few of the duties, whereas frontier fashions rating 0%. This thus represents a job the place human intelligence nonetheless outperforms LLMs.
As we advance, I consider we’ll see fast enhancements in LLM efficiency on ARC AGI 3, although I hope this won’t be the results of benchmark chasing, however moderately the intelligence enchancment of LLMs.
