Apache Flink 2.3 introduces flink-s3-fs-native, a Hadoop-free S3 filesystem plugin built on AWS SDK v2. It replaces the two existing Hadoop-based S3 plugins (hadoop and presto) with a single solution that supports exactly-once sinks, RecoverableWriter, and async NIO I/O via S3TransferManager. Benchmarks show 1.85x faster average checkpoint duration and 2.17x higher throughput vs the Presto plugin, with up to 4.5x speedup at small state sizes. The JAR is ~13 MB vs 30–93 MB for existing plugins, eliminating Hadoop transitive dependencies and CVE exposure from the deprecated AWS SDK v1. Migration requires only swapping the JAR and updating config keys; existing checkpoints remain readable. Per-bucket configuration, AWS CRT client support, and enhanced observability are planned for Flink 2.4.

11m read timeFrom flink.apache.org
Post cover image
Table of contents
Two Plugins, One Filesystem, and No Good Answer #Why This Matters Beyond Engineering #One Stop Solution: Native S3 Filesystem #Performance #Smooth Migration Path #Availability and Roadmap #Get Involved #
6.2K Impressions