AI Snake Oil

Researchers from Princeton conducted 'shadow evaluations' where frontier AI agents were given thousands of dollars in compute and six days to answer open-ended AI research questions from two unpublished papers. Both agent-produced papers were unambiguously rejected by the original authors. Key failure modes included poor judgment leading to premature abandonment of promising directions, underutilization of available resources (less than 50% of API budget spent), inability to creatively respond to feedback, failure to backtrack effectively, and ignoring explicit instructions. The findings challenge optimistic forecasts of recursive self-improvement (RSI) and explosive AI progress, suggesting that hill-climbing on verifiable tasks alone is unlikely to produce broad RSI. The authors apply Amdahl's law to argue that multiple hard-to-resolve bottlenecks could significantly dampen the pace of AI progress even if some components are dramatically accelerated.

AI agents can't yet do open-ended AI research

If you know how LLMs work, the inability to creatively respond to feedback should not be a surprise. Creativity is at the edges and cannot be achieved when you’re regressing to the mean.

Pretty bleak that both papers still got rejected after six days + thousands in compute. The abandon-early and weak feedback failures stack on each other so the agent never stays long enough on a promising line to learn from the rejection signal, and hill-climbing the verifiable tasks just keeps that whole layer untested

Using the original authors as the judges is what makes this evaluation convincing, since it sidesteps the usual problem of a model-judged benchmark rewarding confident-sounding output. The gap between passing narrow coding tasks and doing open-ended research is mostly about knowing which question is worth asking, which no leaderboard really measures. I’d love to see the failure transcripts, because where exactly the agent went wrong is usually more instructive than the pass-or-fail verdict.

I think the unused API budget exposes judgment as the bottleneck, not compute.

The inability to backtrack effectively shows the limits of current agent scaffolding. Right now, long-horizon planning relies heavily on prompt graphs and short-term memory, which causes models to panic or abandon work when hit with minor friction. Until we solve state management across multi-day tasks, giving an agent more compute just means it gets lost faster.