LangChain releases four benchmark environments for evaluating LLM tool use and function calling capabilities. The benchmarks test planning, function composition, and instruction-following across GPT-3.5, GPT-4, Claude 2.1, Mistral 7B, and Mixtral models. Key findings: all models fail on longer trajectories; GPT-4 leads on relational data tasks but surprisingly underperforms GPT-3.5 on Multiverse Math due to pretrained biases; Claude 2.1 is competitive with GPT-4 on 3 of 4 tasks; open-source models struggle with multi-step function composition despite correct single-call formatting. The post also highlights service reliability issues with major providers, including Gemini Pro being excluded due to excessive 5xx errors and overly aggressive safety filters.
