Benchmarking StreamFusion - 2x as fast as Flink?! | Personal Projects With Ex-Google SWE, Ep 32
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A developer building StreamFusion, a project that accelerates Apache Flink SQL jobs by replacing JVM-based operators with Rust and Apache Arrow under the hood, shares benchmarking results showing 2-3x performance gains over stock Flink. The video covers the architecture of replacing Flink's internal operators with columnar, vectorized Rust equivalents, the challenges of efficiently ingesting Kafka data (Avro, Protobuf, JSON) into Arrow buffers using SIMD-accelerated libraries like arrow-avro and simdjson, and the discovery that switching from malloc to Microsoft's mimalloc resolved cross-thread memory allocation bottlenecks when using librdkafka in Rust. Benchmarks run against the Nexmark suite across 20+ SQL queries show consistent 2-3x speedups even when reading row-oriented data from Kafka.