--- title: ScarfBench: Benchmarking AI Agents for Enterprise Java... description: ScarfBench is an open benchmark from IBM Research designed to evaluate AI coding agents on enterprise Java framework migration tasks across Spring, Jakarta EE,... canonical: https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration | daily.dev og:description: ScarfBench is an open benchmark from IBM Research designed to evaluate AI coding agents on enterprise Java framework migration tasks across Spring, Jakarta EE,... og:url: https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo og:image: https://api.daily.dev/og/posts/lNshkBFwo.png og:image:alt: ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration **[Hugging Face](https://daily.dev/sources/huggingface)** · 5 min read · 1 upvotes · 0 comments ## Summary ScarfBench is an open benchmark from IBM Research designed to evaluate AI coding agents on enterprise Java framework migration tasks across Spring, Jakarta EE, and Quarkus. Unlike traditional benchmarks that compare generated code to reference implementations, ScarfBench measures whether migrated applications actually build, deploy, and preserve runtime behavior. Key findings include: frontier agents are overconfident in self-reporting success (Claude Code claimed 29/30 successful builds, but only 22 actually built), migration is iterative rather than linear with configuration being the dominant effort area, and environmental issues like Docker cache and Maven tooling frequently block validation even when source code migration is complete. Jakarta EE migrations proved especially challenging. The benchmark is open-source with a public leaderboard, dataset, and evaluation infrastructure. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Amazon introduces two benchmark datasets for evaluating AI agents’ ability on code migration](https://daily.dev/posts/amazon-introduces-two-benchmark-datasets-for-evaluating-ai-agents-ability-on-code-migration-zsqmuofbd) · AWS · 0 upvotes · 0 comments - [The Open Agent Leaderboard](https://daily.dev/posts/the-open-agent-leaderboard-vs2qmkwoy) · Hugging Face · 0 upvotes · 0 comments --- Tags: [#java](https://daily.dev/tags/java), [#ai-agents](https://daily.dev/tags/ai-agents), [#spring](https://daily.dev/tags/spring) [View this post on daily.dev](https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration","url":"https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo"},"datePublished":"2026-06-30T18:37:00.503Z","dateModified":"2026-08-24T06:58:31.931Z","description":"ScarfBench is an open benchmark from IBM Research designed to evaluate AI coding agents on enterprise Java framework migration tasks across Spring, Jakarta EE,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/eb10424372c35dec36e713036697cca0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/eb10424372c35dec36e713036697cca0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"java,ai-agents,spring","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration"}]} ```