How to Build an AI Agent That Runs its Own LLM Experiments with autoresearch

autoresearch is an open-source tool by Andrej Karpathy that lets an AI agent autonomously run LLM training experiments on a GPU. The agent edits a single training file (train.py), runs 5-minute training sessions, reads the validation bits-per-byte (val_bpb) metric, and decides whether to keep or revert each change — looping indefinitely. The 5-minute wall-clock budget prevents gaming via model size tricks. In Karpathy's depth-12 run, the agent found ~20 stacking improvements over ~700 experiments that transferred to a larger depth-24 model, cutting 'Time to GPT-2' by ~11%. The post covers the repo structure, how val_bpb works, the program.md playbook design, setup steps, and tuning tips for smaller GPUs.