LangChain's annual report analyzes LangSmith usage data across nearly 30k monthly users, revealing key 2024 trends: OpenAI remains the top LLM provider (6x Ollama), while Ollama and Groq gained momentum for open-source model deployment. Vector store rankings stayed stable with Chroma and FAISS leading, though Milvus, MongoDB, and Elastic joined the top 10. LangGraph adoption grew significantly, with 43% of organizations now sending LangGraph traces, and tool-call usage in traces jumped from 0.5% to 21.9%, signaling a shift toward agentic workflows. Average steps per trace rose from 2.8 to 7.7, while LLM calls per trace grew modestly (1.1 to 1.4), indicating more efficient multi-step designs. Annotated runs for human feedback grew 18x year over year, and Python continues to dominate LangSmith SDK usage (84.7%) though JavaScript usage tripled to 15.3%.
Table of contents
Infrastructure usageBuilding with LangChain productsPerformance and optimizationLLM testing & evaluationConclusionQuestions this post answers
What percentage of LangSmith organizations are using LangGraph for agent workflows?
43% of LangSmith organizations send LangGraph traces, up from its initial release in March 2024. These traces represent complex, orchestrated multi-step tasks that go beyond simple LLM interactions, reflecting the broader shift toward agentic AI application design observed across LangSmith's user base during the year. Developers evaluating agent frameworks can track adoption trends like this on daily.dev.
How much did tool calling increase in LLM applications during 2024 according to LangSmith data?
Tool calls appeared in an average of 21.9% of LangSmith traces in 2024, up sharply from just 0.5% in 2023. Tool calling lets a model autonomously invoke functions or external resources, such as writing to databases, and its rise reflects growing agentic behavior where models decide when to take action. Anyone building agentic LLM systems can follow shifts like this via daily.dev.
Which LLM provider is most used by developers building applications with LangSmith?
OpenAI remains the most-used LLM provider among LangSmith users, used more than 6 times as much as Ollama, the second-most popular provider. Ollama and Groq both gained significant momentum during the year, breaking into the top 5 as developers sought more flexible open-source model deployment options for local and cloud use. Teams weighing LLM provider choices can keep tabs on shifts like this through daily.dev.