LangChain benchmarks how a single ReAct agent's performance degrades as it's given more domains (instructions + tools). Testing claude-3.5-sonnet, gpt-4o, o1, o3-mini, and llama-3.3-70B on calendar scheduling and customer support tasks, key findings show: more context and more tools consistently hurt performance; longer tool-calling trajectories degrade faster; o1, o3-mini, and claude-3.5-sonnet significantly outperform gpt-4o and llama-3.3-70B; o3-mini matches top models with small context but drops sharply as context grows; claude-3.5-sonnet is more stable under large context for complex tool-use tasks. The study sets the stage for follow-up benchmarks comparing single-agent vs. multi-agent architectures.
Table of contents
Problem Statement and TerminologyExperiment DesignCalendar Scheduling TasksCustomer Support TasksPerformance across Trajectory LengthsGeneral Trends3 Impressions