AlignEval is a web app designed to simplify building LLM-based evaluators through a four-step workflow: upload data, label samples with binary pass/fail decisions, write task-specific evaluation criteria, and run automated optimization. The core philosophy is to work backward from actual data rather than crafting criteria in the abstract. The app uses gpt-4o-mini and claude-3-haiku for evaluation, supports optimization via dev/test splits, and is built with Next.js (TypeScript) on the frontend and Python/FastAPI on the backend, hosted on Railway. Key insights include the value of binary labeling for accuracy and speed, the danger of generic out-of-the-box eval criteria, and the importance of labeling at least 50 samples before writing criteria. The author also shares the full development journey including framework selection, hosting decisions, and a detailed to-do list.
