<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/what-is-the-best-harness-for-each-model--7w7naxkvs" -->

---
title: What is the best harness for each model? | daily.dev
description: Switching your model should make you reconsider your coding-agent harness too, a 27,280-run benchmark finds. FrontierHarness Eval v1.1 tested 22 harness...
canonical: https://daily.dev/posts/what-is-the-best-harness-for-each-model--7w7naxkvs
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: What is the best harness for each model? | daily.dev
og:description: Switching your model should make you reconsider your coding-agent harness too, a 27,280-run benchmark finds. FrontierHarness Eval v1.1 tested 22 harness...
og:url: https://daily.dev/posts/what-is-the-best-harness-for-each-model--7w7naxkvs
og:image: https://api.daily.dev/og/posts/7w7naXKvs.png
og:image:alt: What is the best harness for each model?
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# What is the best harness for each model?

**[Hacker News](https://daily.dev/sources/hn)** · 25 min read · 0 upvotes · 0 comments

## Summary

Switching your model should make you reconsider your coding-agent harness too, a 27,280-run benchmark finds. FrontierHarness Eval v1.1 tested 22 harness configurations across 5 models on 62 tasks, 4 attempts each. Each model had a different best-value harness: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek, Oh My Pi with Muse and Alma with Kimi-K3. Codex CLI went from 1st in cost with GLM-5.3 to 16th with Kimi-K3. The model explains 92.8% of cost variance; with the model fixed, the priciest harness still cost 1.8 to 5.0 times more per task than the cheapest. Higher cost didn't mean a higher pass rate, and 64.5% of spending went to trials that failed.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://runta.com/blog/introducing-frontierharness-eval-v1-1>

## Questions this post answers

### Does switching to a cheaper or different LLM mean I should also switch my coding agent harness?

Yes, the best-value harness changes depending on the model. In a 27,280-run benchmark across 22 harnesses and 5 models, Codex CLI was the cheapest-per-task harness with GLM-5.3 but ranked 16th with Kimi-K3. Each of the five tested models had a different top harness, and cost ranks shifted by a median of 11 out of 22 positions between models.

_Developers tuning agent setups after a model switch can track these harness benchmarks on daily.dev._

### Does a more expensive coding agent harness actually produce a higher task pass rate?

No, a higher cost does not reliably buy a higher pass rate. Across the five tested models (GLM-5.3, GLM Flash, DeepSeek, Muse, Kimi-K3), the rank correlation between cost per task and pass rate ranged from -0.39 to +0.22. With four of the five models, the most expensive harness per pass actually passed less often than the cheapest one.

_Anyone weighing agent cost against reliability can follow harness benchmarking results on daily.dev._

### Why do some AI coding agent harnesses cost more per task than others even with the same model?

The number of steps in the agent loop drives the cost difference, not prompt size, tool count, or cache hit rate. Across 22 harnesses, steps per pass correlated with cost per pass at +0.51, while start tokens, tools offered, and cache hit rate all showed near-zero correlation. Each step resends the full context, so more steps means more repeated input tokens billed.

_Engineers optimizing agent spend can dig into these cost-driver findings on daily.dev._

## Community take

How the wider developer community reacted, aggregated from 1 discussion and 114 comments across hackernews (as of 2026-11-17).

**TL;DR:** Commenters largely debate the methodology and framing of the benchmark rather than its headline finding, with heavy scrutiny of task selection, reproducibility, and whether 'cost per pass' is even the right metric, while others question if harness choice matters as much as claimed.

**Sentiment:** 10% positive · 35% mixed · 55% skeptical

**The case for**

- Some found it valuable that tokens are being spent to evaluate harness capabilities systematically.
- A few appreciated the author (identified as the project's founder) directly engaging and answering questions in the thread.
- Several agreed that switching harness when switching models is a real, previously observed phenomenon worth highlighting.

**The pushback**

- Many questioned how the specific tasks were selected, suspecting cherry-picking since the methodology wasn't fully documented.
- Several argued the post/title was edited to be more sensational ('64% of spend went to failing runs') than the original, more neutral title.
- Multiple commenters doubted the benchmark's real-world applicability since pass/fail on tests doesn't capture code quality or user steering.
- Some accused the poster of using the thread primarily for self-promotion rather than genuine discussion.
- Commenters noted high run-to-run variance and small sample size (4 attempts per task) undermine confidence in the conclusions.
- Some felt the distinction between 'harness' and 'agent' is blurry or meaningless in this context.
- A few called the result an unsurprising or even tautological finding dressed up as novel insight.

**By community**

- hackernews (mixed): Discussion splits between methodological skepticism (task selection, reproducibility, editorialized title, promotional concerns) and broader tangential debates about coding agents, harnesses, and the future of software development.

**Hottest debate:** Whether the benchmark's task selection and small sample size make its headline conclusions (cost variance, 64% wasted spend) meaningful or just cherry-picked noise.

**Open questions**

- How exactly were the specific benchmark tasks chosen, and why only a subset of the full terminal-bench dataset?
- Would the results hold up under a second independent sample given the high run-to-run variance?
- Does 'cost per pass' meaningfully differ from 'cost per successful task' in a way that changes which harness looks best?
- Why were certain popular harnesses/agents (e.g. Kilo Code) excluded from the comparison?
- Does harness choice matter as much for closed models (ChatGPT, Claude) as it does for the open-weight models tested here?

**Highlights**

> The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers. It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes. Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".
> — [seizethecheese on hackernews · 6 comments](https://news.ycombinator.com/item?id=50533954)

> Okay, I wanted to know “which tasks” and dig into the methodology, and its documented nowhere! A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.
> — [seizethecheese on hackernews · 4 comments](https://news.ycombinator.com/item?id=50534140)

> "In 27,280 coding agent runs, 64% of inference spend went to runs that failed" (the title this submission was submitted with) is a pretty editorialized title. The page's title is: "Introducing FrontierHarness Eval v1.1"
> — [jrmg on hackernews](https://news.ycombinator.com/item?id=50533768)

> From the Hacker News Guidelines [1]: >  Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity. Your post history is almost entirely promotion of your product. Please stop. I'm flagging your post. [1]: https://news.ycombinator.com/newsguidelines.html
> — [markasoftware on hackernews](https://news.ycombinator.com/item?id=50537053)

> I don't like this framing. For a given model, the best harness depends on the problem. As models become increasingly capable, the harness  appears to matter more than the model. This is perhaps controversial but allow me to explain. Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times. This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct. But I can't build a mobile app with this harness, or make a movie. These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do. Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice. Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.
> — [xyzzy\_plugh on hackernews · 2 comments](https://news.ycombinator.com/item?id=50535332)

> Hi Guanlan, I vouched for your comment since you involved in this post, but you should be aware that all of your posts have been flagged, likely for AI patterns in your writing. > Don't post generated text or AI-edited text. HN is for conversation between humans. https://news.ycombinator.com/newsguidelines.html
> — [radlad on hackernews](https://news.ycombinator.com/item?id=50533068)

**Source threads**

- [hackernews](https://news.ycombinator.com/item?id=50532191) · 303 points · 114 comments

## Similar posts on daily.dev

- [Are Harnesses Worth the Cost?](https://daily.dev/posts/are-harnesses-worth-the-cost--hrgnwzebs) · Martin Fowler · 0 upvotes · 0 comments
- [Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase](https://daily.dev/posts/benchmarking-coding-agents-on-databricks-multi-million-line-codebase-qjzvv6o07) · databricks · 1 upvotes · 0 comments

---

Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#openai-codex](https://daily.dev/tags/openai-codex)

[View this post on daily.dev](https://daily.dev/posts/what-is-the-best-harness-for-each-model--7w7naxkvs)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"What is the best harness for each model?","url":"https://daily.dev/posts/what-is-the-best-harness-for-each-model--7w7naxkvs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/what-is-the-best-harness-for-each-model--7w7naxkvs"},"datePublished":"2026-11-17T17:26:14.469Z","dateModified":"2026-11-17T22:50:45.568Z","description":"Switching your model should make you reconsider your coding-agent harness too, a 27,280-run benchmark finds. FrontierHarness Eval v1.1 tested 22 harness...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4c0bf175bd43b8236f60bbe69a14615b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4c0bf175bd43b8236f60bbe69a14615b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/what-is-the-best-harness-for-each-model--7w7naxkvs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,openai-codex","timeRequired":"PT25M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"What is the best harness for each model?"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/what-is-the-best-harness-for-each-model--7w7naxkvs#faq","mainEntity":[{"@type":"Question","name":"Does switching to a cheaper or different LLM mean I should also switch my coding agent harness?","acceptedAnswer":{"@type":"Answer","text":"Yes, the best-value harness changes depending on the model. In a 27,280-run benchmark across 22 harnesses and 5 models, Codex CLI was the cheapest-per-task harness with GLM-5.3 but ranked 16th with Kimi-K3. Each of the five tested models had a different top harness, and cost ranks shifted by a median of 11 out of 22 positions between models. Developers tuning agent setups after a model switch can track these harness benchmarks on daily.dev."}},{"@type":"Question","name":"Does a more expensive coding agent harness actually produce a higher task pass rate?","acceptedAnswer":{"@type":"Answer","text":"No, a higher cost does not reliably buy a higher pass rate. Across the five tested models (GLM-5.3, GLM Flash, DeepSeek, Muse, Kimi-K3), the rank correlation between cost per task and pass rate ranged from -0.39 to +0.22. With four of the five models, the most expensive harness per pass actually passed less often than the cheapest one. Anyone weighing agent cost against reliability can follow harness benchmarking results on daily.dev."}},{"@type":"Question","name":"Why do some AI coding agent harnesses cost more per task than others even with the same model?","acceptedAnswer":{"@type":"Answer","text":"The number of steps in the agent loop drives the cost difference, not prompt size, tool count, or cache hit rate. Across 22 harnesses, steps per pass correlated with cost per pass at +0.51, while start tokens, tools offered, and cache hit rate all showed near-zero correlation. Each step resends the full context, so more steps means more repeated input tokens billed. Engineers optimizing agent spend can dig into these cost-driver findings on daily.dev."}}]}
```

