A walkthrough of using LangSmith to evaluate and compare fine-tuned open source LLMs (Llama2-7b and Llama2-13b) against GPT-3.5-turbo on a SQL generation task. The process covers fine-tuning both models on a sql-create-context dataset via Replicate, uploading a 1000-row validation set to LangSmith, and running automated chain-of-thought QA evaluations using GPT-4 as the judge. Key findings: the 13b model fine-tuned on 10k rows nearly matched GPT-3.5-turbo accuracy, while the 7b model fine-tuned on 78k rows showed strong accuracy with better response times. The post includes Python code snippets for dataset creation, fine-tuning via Replicate, and running evaluations in LangSmith.
2 Impressions