Chess serves as a structured, stateful environment for evaluating AI model behavior beyond static benchmarks. Using Dagster for orchestration and aisuite for a unified model interface, the author runs repeatable tournaments pitting random agents, Stockfish, GPT-4o, GPT-3.5-turbo, and Claude against each other. Key findings: general-purpose LLMs perform no better than random play against Stockfish, and when LLMs face each other, games consistently end in fivefold repetition draws rather than decisive results. This reveals that safety-optimized LLMs are biased toward avoiding loss over seeking wins, preferring stagnation as a locally safe strategy. The experiment demonstrates how stateful, adversarial simulations expose failure modes — like repetition loops and risk aversion — that standard benchmarks miss.
