Researchers from Tufa Labs discuss ARC-AGI-3, a benchmark that challenges AI agents to infer game goals and rules from raw pixel observations with no instructions. Unlike earlier ARC versions, ARC-AGI-3 introduces dynamic goal acquisition and action efficiency scoring, making brute-force approaches ineffective. The team shares their competition experience: an early brute-force stochastic search won the preview round but failed when the benchmark was hardened. Their current approach uses LLM-based coding agents that write Python harnesses, leverage built-in game priors (e.g., maze recognition), and balance exploration vs. exploitation. Key findings include: frontier models can solve 50–66% of training games but inefficiently; the 36% leaderboard score reflects action efficiency penalties, not raw solve rate; long-context consistency over hundreds of thousands of tokens is a major bottleneck; and LLMs exhibit a recurring failure mode of locking onto wrong hypotheses and failing to shift abstraction levels. The conversation also covers broader questions about whether LLMs perform genuine planning or simulate it, the role of core knowledge priors vs. learned abstractions, and how requirements-based engineering helps teams work effectively with AI coding agents.

1h 24m watch time
148 Impressions