--- title: ScarfBench: Benchmarking AI Agents for Enterprise Java... description: ScarfBench is an open benchmark from IBM Research designed to evaluate AI coding agents on enterprise Java framework migration tasks across Spring, Jakarta EE,... canonical: https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration | daily.dev og:description: ScarfBench is an open benchmark from IBM Research designed to evaluate AI coding agents on enterprise Java framework migration tasks across Spring, Jakarta EE,... og:url: https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo og:image: https://api.daily.dev/og/posts/lNshkBFwo.png og:image:alt: ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration **[Hugging Face](https://daily.dev/sources/huggingface)** · 5 min read · 1 upvotes · 0 comments ## Summary ScarfBench is an open benchmark from IBM Research designed to evaluate AI coding agents on enterprise Java framework migration tasks across Spring, Jakarta EE, and Quarkus. Unlike traditional benchmarks that compare generated code to reference implementations, ScarfBench measures whether migrated applications actually build, deploy, and preserve runtime behavior. Key findings include: frontier agents are overconfident in self-reporting success (Claude Code claimed 29/30 successful builds, but only 22 actually built), migration is iterative rather than linear with configuration being the dominant effort area, and environmental issues like Docker cache and Maven tooling frequently block validation even when source code migration is complete. Jakarta EE migrations proved especially challenging. The benchmark is open-source with a public leaderboard, dataset, and evaluation infrastructure. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How accurate is Claude Code at self-reporting successful builds when migrating Java applications between frameworks? Claude Code overreported success: it claimed 29 of 30 whole Java applications built successfully after framework migration, but independent verification showed only 22 actually built. The one application it marked as failed actually built correctly, indicating agent self-assessment is not a reliable signal and independent build and test validation remains necessary. _Anyone evaluating AI agents for framework migration should track independent benchmarks like this on daily.dev before trusting self-reported success._ ### Why is migrating a Java application from Spring to Jakarta EE difficult for AI coding agents? Migration difficulty stems less from code translation and more from managing dependencies across configuration, infrastructure, and runtime environments. Configuration, web, database, and service layers are the most frequently revisited during migration, with common transitions between configuration/web and service/database, showing migration is an iterative dependency-resolution process rather than a linear source-to-source transformation. Jakarta EE proved particularly challenging among target frameworks. _Teams weighing AI-assisted Java modernization can follow benchmark results on daily.dev to gauge readiness before committing to a migration._ ### What non-code issues commonly block AI agents from completing enterprise Java framework migrations? Agents frequently struggled with environmental and tooling problems rather than source code itself, including Docker cache inconsistencies, port connectivity issues, and Maven wrapper or build tooling failures. These operational concerns often delayed validation even when the underlying source-code migration was largely complete, showing that reliable build and deployment infrastructure matters as much as code translation accuracy. _Developers debugging agent-driven migrations can compare failure patterns like these on daily.dev before automating the process further._ ## Similar posts on daily.dev - [Amazon introduces two benchmark datasets for evaluating AI agents’ ability on code migration](https://daily.dev/posts/amazon-introduces-two-benchmark-datasets-for-evaluating-ai-agents-ability-on-code-migration-zsqmuofbd) · AWS · 0 upvotes · 0 comments - [The Open Agent Leaderboard](https://daily.dev/posts/the-open-agent-leaderboard-vs2qmkwoy) · Hugging Face · 0 upvotes · 0 comments - [Transform your AI coding agent into a deterministic Java Spring expert](https://daily.dev/posts/transform-your-ai-coding-agent-into-a-deterministic-java-spring-expert-n8wcjz7q9) · The New Stack · 0 upvotes · 0 comments - [Introducing the Kotlin Benchmark for AI Coding Agents](https://daily.dev/posts/introducing-the-kotlin-benchmark-for-ai-coding-agents-5e9vdjnes) · JetBrains · 1 upvotes · 0 comments - [Introducing o11y-bench: an open benchmark for AI agents running observability workflows](https://daily.dev/posts/introducing-o11y-bench-an-open-benchmark-for-ai-agents-running-observability-workflows-psucsa908) · Grafana Labs · 20 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#java](https://daily.dev/tags/java), [#spring](https://daily.dev/tags/spring) [View this post on daily.dev](https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration","url":"https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo"},"datePublished":"2026-06-30T18:37:00.503Z","dateModified":"2026-09-14T06:47:19.915Z","description":"ScarfBench is an open benchmark from IBM Research designed to evaluate AI coding agents on enterprise Java framework migration tasks across Spring, Jakarta EE,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/eb10424372c35dec36e713036697cca0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/eb10424372c35dec36e713036697cca0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,java,spring","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/scarfbench-benchmarking-ai-agents-for-enterprise-java-framework-migration-lnshkbfwo#faq","mainEntity":[{"@type":"Question","name":"How accurate is Claude Code at self-reporting successful builds when migrating Java applications between frameworks?","acceptedAnswer":{"@type":"Answer","text":"Claude Code overreported success: it claimed 29 of 30 whole Java applications built successfully after framework migration, but independent verification showed only 22 actually built. The one application it marked as failed actually built correctly, indicating agent self-assessment is not a reliable signal and independent build and test validation remains necessary. Anyone evaluating AI agents for framework migration should track independent benchmarks like this on daily.dev before trusting self-reported success."}},{"@type":"Question","name":"Why is migrating a Java application from Spring to Jakarta EE difficult for AI coding agents?","acceptedAnswer":{"@type":"Answer","text":"Migration difficulty stems less from code translation and more from managing dependencies across configuration, infrastructure, and runtime environments. Configuration, web, database, and service layers are the most frequently revisited during migration, with common transitions between configuration/web and service/database, showing migration is an iterative dependency-resolution process rather than a linear source-to-source transformation. Jakarta EE proved particularly challenging among target frameworks. Teams weighing AI-assisted Java modernization can follow benchmark results on daily.dev to gauge readiness before committing to a migration."}},{"@type":"Question","name":"What non-code issues commonly block AI agents from completing enterprise Java framework migrations?","acceptedAnswer":{"@type":"Answer","text":"Agents frequently struggled with environmental and tooling problems rather than source code itself, including Docker cache inconsistencies, port connectivity issues, and Maven wrapper or build tooling failures. These operational concerns often delayed validation even when the underlying source-code migration was largely complete, showing that reliable build and deployment infrastructure matters as much as code translation accuracy. Developers debugging agent-driven migrations can compare failure patterns like these on daily.dev before automating the process further."}}]} ```