--- title: How To Benchmark AI Models Yourself | daily.dev description: A hands-on walkthrough demonstrates how to benchmark AI models locally, covering hosting models with vLLM and Ollama, then running evaluations using the LM... canonical: https://daily.dev/posts/how-to-benchmark-ai-models-yourself-rcabt5sax twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How To Benchmark AI Models Yourself | daily.dev og:description: A hands-on walkthrough demonstrates how to benchmark AI models locally, covering hosting models with vLLM and Ollama, then running evaluations using the LM... og:url: https://daily.dev/posts/how-to-benchmark-ai-models-yourself-rcabt5sax og:image: https://api.daily.dev/og/posts/RcaBT5SAx.png og:image:alt: How To Benchmark AI Models Yourself og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How To Benchmark AI Models Yourself **[NeuralNine](https://daily.dev/sources/neuralnine)** · 20 min read · 1 upvotes · 0 comments ## Summary A hands-on walkthrough demonstrates how to benchmark AI models locally, covering hosting models with vLLM and Ollama, then running evaluations using the LM Evaluation Harness (for tasks like Hellaswag and ARC Challenge), Eval Plus (for Human Eval), and Harbor (for Terminal Bench). It also shows how to create a custom benchmark from scratch, including a JSONL dataset and YAML config, and evaluate small models like Qwen 2.5 Coder 0.5B and Qwen 3 0.6B against it. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I run the LM Evaluation Harness against a model hosted locally with vLLM? Install lm_eval with the hf and vllm extras via 'uv add lm_eval[hf,vllm,api]', then run it with --model local-completions and model_args specifying the model identifier, base_url pointing to the vLLM endpoint's v1/completions path, tokenizer, tokenizer_backend=huggingface, and tokenized_requests=false. Specify --tasks (e.g. hellaswag or arc_challenge) and optionally --limit to test a subset before running the full evaluation. _Anyone setting up local model evaluation pipelines can find configuration details like these curated on daily.dev._ ### How do I evaluate a model on Terminal-Bench using Ollama? Install Harbor via 'uv add harbor', make sure Ollama is running and the target model is pulled, then run 'harbor run -D terminal-bench/terminal-bench-2 -M "ollama_chat/" --api_base http://localhost:11434'. Use -T instead of -D with a specific task name to test a single task instead of the full benchmark, since the full run can take hours. _Teams comparing agent harnesses for terminal-based coding tasks track workflow guides like this on daily.dev._ ### How can I create and run my own custom benchmark with the LM Evaluation Harness? Define the benchmark as a JSONL file with examples and answer indices plus a matching YAML config file (must use the .yaml extension, not .yml) describing task settings like multiple-choice accuracy. Then run lm_eval with --tasks set to the custom benchmark name and --include_path pointing to the directory containing these files, alongside the usual model and endpoint arguments. _Developers designing evaluation sets for niche or fine-tuned models can reference setup steps like these via daily.dev._ ## Similar posts on daily.dev - [Self-Hosting AI Code Review: Local Models for Better Code Quality](https://daily.dev/posts/self-hosting-ai-code-review-local-models-for-better-code-quality-ktiohtydz) · SitePoint · 0 upvotes · 0 comments - [The Complete Guide to Running AI Locally: Everything You Need to Know Before You Start](https://daily.dev/posts/the-complete-guide-to-running-ai-locally-everything-you-need-to-know-before-you-start-3weeeznf1) · Medium · 0 upvotes · 0 comments - [Self-Hosting Your First LLM](https://daily.dev/posts/self-hosting-your-first-llm-13jpheaxf) · Towards Data Science · 1 upvotes · 0 comments - [Run a Local AI Model with Ollama in 15 Minutes](https://daily.dev/posts/run-a-local-ai-model-with-ollama-in-15-minutes-5wx17jt3x) · Machine Learning Mastery · 1 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ollama](https://daily.dev/tags/ollama), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/how-to-benchmark-ai-models-yourself-rcabt5sax) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How To Benchmark AI Models Yourself","url":"https://daily.dev/posts/how-to-benchmark-ai-models-yourself-rcabt5sax","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-to-benchmark-ai-models-yourself-rcabt5sax"},"datePublished":"2026-08-21T16:12:56.204Z","dateModified":"2026-09-14T06:06:13.963Z","description":"A hands-on walkthrough demonstrates how to benchmark AI models locally, covering hosting models with vLLM and Ollama, then running evaluations using the LM...","image":"https://i.ytimg.com/vi/fDx_C1TF7f8/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/fDx_C1TF7f8/sddefault.jpg","isAccessibleForFree":true,"articleSection":"NeuralNine","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"NeuralNine","logo":"https://media.daily.dev/image/upload/s--bMYO07OB--/f_auto,q_auto/v1777800757/logos/neuralnine?_a=BAMAMiWQ0","url":"https://daily.dev/sources/neuralnine"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-to-benchmark-ai-models-yourself-rcabt5sax","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,ollama,vllm","timeRequired":"PT20M","video":{"@type":"VideoObject","name":"How To Benchmark AI Models Yourself","description":"A hands-on walkthrough demonstrates how to benchmark AI models locally, covering hosting models with vLLM and Ollama, then running evaluations using the LM...","thumbnailUrl":"https://i.ytimg.com/vi/fDx_C1TF7f8/sddefault.jpg","uploadDate":"2026-08-21T16:12:56.204Z","duration":"PT20M","url":"https://api.daily.dev/r/RcaBT5SAx","embedUrl":"https://www.youtube.com/embed/fDx_C1TF7f8"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"NeuralNine","item":"https://daily.dev/sources/neuralnine"},{"@type":"ListItem","position":3,"name":"How To Benchmark AI Models Yourself"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-to-benchmark-ai-models-yourself-rcabt5sax#faq","mainEntity":[{"@type":"Question","name":"How do I run the LM Evaluation Harness against a model hosted locally with vLLM?","acceptedAnswer":{"@type":"Answer","text":"Install lm_eval with the hf and vllm extras via 'uv add lm_eval[hf,vllm,api]', then run it with --model local-completions and model_args specifying the model identifier, base_url pointing to the vLLM endpoint's v1/completions path, tokenizer, tokenizer_backend=huggingface, and tokenized_requests=false. Specify --tasks (e.g. hellaswag or arc_challenge) and optionally --limit to test a subset before running the full evaluation. Anyone setting up local model evaluation pipelines can find configuration details like these curated on daily.dev."}},{"@type":"Question","name":"How do I evaluate a model on Terminal-Bench using Ollama?","acceptedAnswer":{"@type":"Answer","text":"Install Harbor via 'uv add harbor', make sure Ollama is running and the target model is pulled, then run 'harbor run -D terminal-bench/terminal-bench-2 -M \"ollama_chat/\" --api_base http://localhost:11434'. Use -T instead of -D with a specific task name to test a single task instead of the full benchmark, since the full run can take hours. Teams comparing agent harnesses for terminal-based coding tasks track workflow guides like this on daily.dev."}},{"@type":"Question","name":"How can I create and run my own custom benchmark with the LM Evaluation Harness?","acceptedAnswer":{"@type":"Answer","text":"Define the benchmark as a JSONL file with examples and answer indices plus a matching YAML config file (must use the .yaml extension, not .yml) describing task settings like multiple-choice accuracy. Then run lm_eval with --tasks set to the custom benchmark name and --include_path pointing to the directory containing these files, alongside the usual model and endpoint arguments. Developers designing evaluation sets for niche or fine-tuned models can reference setup steps like these via daily.dev."}}]} ```