Testing AI prompts and comparing models with promptfoo

Prompts in AI-integrated applications can produce unexpected results, and manual testing doesn't scale. Promptfoo automates prompt evaluation by running test cases against configured models and checking responses with deterministic assertions (equals, contains, JSON schema, cost/latency) and model-graded assertions (tone, relevance, factual consistency). A practical example shows extracting structured JSON from customer messages, comparing two OpenAI GPT tiers side by side. Configurations can be written in YAML or TypeScript, prompts and fixtures can be loaded from files, and tasks can be organized by feature directory. Results are viewable in a terminal table or a local web UI showing outputs, assertion results, token usage, and cost. Evaluations can also run in CI/CD pipelines to catch regressions before deployment. Alternatives like Langfuse and Microsoft.Extensions.AI.Evaluation are briefly noted.