This post discusses the challenges of building reliable LLM products and how Dosu uses evaluation driven development (EDD) to iterate and improve its product. It also highlights the upgrade of Dosu's LLM monitoring stack with LangSmith and the benefits it provides. The post mentions the use of LangSmith's SDK, finding failure modes, and automating evaluation dataset collection. Dosu is hiring.
Table of contents
What is Dosu?Early DaysEvaluation Driven DevelopmentKeeping the Quality Bar High at ScaleImplementing LangSmith via the SDKFinding FailuresAutomating Evaluation Dataset Collection1 Impression