A practical guide to evaluating LLMs using EleutherAI's Evaluation Harness against OpenAI-compatible API endpoints served by Hugging Face TGI or vLLM. The tutorial walks through deploying Llama 3.1 8B Instruct with TGI on an AWS GPU instance, then running lm_eval with the local-chat-completions interface against IFEval and GSM8K (Chain of Thought) benchmarks. Results closely match Meta's officially reported numbers (IFEval: 0.803 vs 0.804, GSM8K: 0.856 vs 0.845), validating both the model and the production serving setup.
1 Impression