Factory AI, a company building autonomous Droids for SDLC automation, adopted self-hosted LangSmith to solve observability and feedback loop challenges in their LLM pipelines. By integrating LangSmith with AWS CloudWatch, they gained precise tracing across agentic pipeline stages and could debug context-awareness issues like hallucinations tied directly to individual LLM calls. Using LangSmith's Feedback API, Factory automated prompt optimization by collecting real customer feedback, exporting it to datasets, and using LLMs to analyze why certain prompts produced poor outputs. This replaced manual, time-consuming iteration and resulted in a 2x improvement in iteration speed. Customers saw ~20% reduction in open-to-merge time and a 3x reduction in code churn within 90 days. Factory has saved over 550,000 hours of development time across clients and recently raised $15M in Series A funding led by Sequoia Capital.
