--- title: Managing Flink Operator State In StreamFusion! |... description: A walkthrough of how Flink's raw keyed state mechanism enables proper state management and autoscaling for native (Rust-based) operators in a streaming system... canonical: https://daily.dev/posts/managing-flink-operator-state-in-streamfusion-personal-projects-with-ex-google-swe-ep-33-cmtafqcwk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Managing Flink Operator State In StreamFusion! | Personal Projects With Ex-Google SWE, Ep 33 | daily.dev og:description: A walkthrough of how Flink's raw keyed state mechanism enables proper state management and autoscaling for native (Rust-based) operators in a streaming system... og:url: https://daily.dev/posts/managing-flink-operator-state-in-streamfusion-personal-projects-with-ex-google-swe-ep-33-cmtafqcwk og:image: https://api.daily.dev/og/posts/cmtAFQcWK.png og:image:alt: Managing Flink Operator State In StreamFusion! | Personal Projects With Ex-Google SWE, Ep 33 og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Managing Flink Operator State In StreamFusion! | Personal Projects With Ex-Google SWE, Ep 33 **[Jordan has no life](https://daily.dev/sources/jordan-has-no-life)** · 7 min read · 0 upvotes · 0 comments ## Summary A walkthrough of how Flink's raw keyed state mechanism enables proper state management and autoscaling for native (Rust-based) operators in a streaming system called StreamFusion. The core problem: when Rust operators checkpoint state as opaque byte arrays, Flink can't redistribute that state during rescaling. The solution is using Flink's raw keyed state API, which lets operators associate byte arrays with specific key groups — fixed hash-range partitions of the key space. This allows Flink to route state blobs to the correct operator instances during autoscaling without needing to understand the internal state structure, keeping JNI overhead minimal while preserving Flink's fault-tolerance semantics. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Apache Flink support autoscaling when operator state is managed outside the JVM, like in Rust? Flink's raw keyed state API lets a native operator report its serialized state as an opaque byte array tagged with a key group, so Flink can redistribute the correct blobs to operators when parallelism changes without needing to understand the blob's internal structure. This avoids JNI calls into the state store during normal operation, keeping Rust-managed state fast while still supporting Flink's checkpoint and rescale mechanics. _daily.dev surfaces engineering deep dives like this for teams combining Flink with native-language operators._ ### Why did checkpointing a raw byte array from a Rust-managed Flink operator break autoscaling? Flink had no way to know which subset of keys the byte array corresponded to, so when parallelism changed (say from three operator instances to four) it could not determine how to split and redistribute the checkpointed state across the new set of operators, causing the job to fail on restart. Tagging each blob with its key group solves this by letting Flink route key groups correctly regardless of internal state format. _following practical accounts of streaming state bugs on daily.dev helps before autoscaling surprises hit production._ ## Similar posts on daily.dev - [Why We Ditched Flink Table API Joins: Cutting State by 75% with DataStream Unions](https://daily.dev/posts/why-we-ditched-flink-table-api-joins-cutting-state-by-75-with-datastream-unions-ynzazjzo0) · Zalando · 0 upvotes · 0 comments - [Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator](https://daily.dev/posts/lyft-moves-streaming-fleet-to-apache-flink-kubernetes-operator-jtednll3i) · InfoQ · 0 upvotes · 0 comments --- Tags: [#rust](https://daily.dev/tags/rust), [#apache-flink](https://daily.dev/tags/apache-flink) [View this post on daily.dev](https://daily.dev/posts/managing-flink-operator-state-in-streamfusion-personal-projects-with-ex-google-swe-ep-33-cmtafqcwk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Managing Flink Operator State In StreamFusion! | Personal Projects With Ex-Google SWE, Ep 33","url":"https://daily.dev/posts/managing-flink-operator-state-in-streamfusion-personal-projects-with-ex-google-swe-ep-33-cmtafqcwk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/managing-flink-operator-state-in-streamfusion-personal-projects-with-ex-google-swe-ep-33-cmtafqcwk"},"datePublished":"2026-07-11T18:21:35.758Z","dateModified":"2026-09-13T21:41:01.611Z","description":"A walkthrough of how Flink's raw keyed state mechanism enables proper state management and autoscaling for native (Rust-based) operators in a streaming system...","image":"https://i.ytimg.com/vi/MO-7fFjIC3s/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/MO-7fFjIC3s/sddefault.jpg","isAccessibleForFree":true,"articleSection":"Jordan has no life","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Jordan has no life","logo":"https://media.daily.dev/image/upload/s--JskHVquu--/f_auto,q_auto/v1780213743/logos/jordan-has-no-life?_a=BAMAMiWQ0","url":"https://daily.dev/sources/jordan-has-no-life"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/managing-flink-operator-state-in-streamfusion-personal-projects-with-ex-google-swe-ep-33-cmtafqcwk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"rust,apache-flink","timeRequired":"PT7M","video":{"@type":"VideoObject","name":"Managing Flink Operator State In StreamFusion! | Personal Projects With Ex-Google SWE, Ep 33","description":"A walkthrough of how Flink's raw keyed state mechanism enables proper state management and autoscaling for native (Rust-based) operators in a streaming system...","thumbnailUrl":"https://i.ytimg.com/vi/MO-7fFjIC3s/sddefault.jpg","uploadDate":"2026-07-11T18:21:35.758Z","duration":"PT7M","url":"https://api.daily.dev/r/cmtAFQcWK","embedUrl":"https://www.youtube.com/embed/MO-7fFjIC3s"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Jordan has no life","item":"https://daily.dev/sources/jordan-has-no-life"},{"@type":"ListItem","position":3,"name":"Managing Flink Operator State In StreamFusion! | Personal Projects With Ex-Google SWE, Ep 33"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/managing-flink-operator-state-in-streamfusion-personal-projects-with-ex-google-swe-ep-33-cmtafqcwk#faq","mainEntity":[{"@type":"Question","name":"How does Apache Flink support autoscaling when operator state is managed outside the JVM, like in Rust?","acceptedAnswer":{"@type":"Answer","text":"Flink's raw keyed state API lets a native operator report its serialized state as an opaque byte array tagged with a key group, so Flink can redistribute the correct blobs to operators when parallelism changes without needing to understand the blob's internal structure. This avoids JNI calls into the state store during normal operation, keeping Rust-managed state fast while still supporting Flink's checkpoint and rescale mechanics. daily.dev surfaces engineering deep dives like this for teams combining Flink with native-language operators."}},{"@type":"Question","name":"Why did checkpointing a raw byte array from a Rust-managed Flink operator break autoscaling?","acceptedAnswer":{"@type":"Answer","text":"Flink had no way to know which subset of keys the byte array corresponded to, so when parallelism changed (say from three operator instances to four) it could not determine how to split and redistribute the checkpointed state across the new set of operators, causing the job to fail on restart. Tagging each blob with its key group solves this by letting Flink route key groups correctly regardless of internal state format. following practical accounts of streaming state bugs on daily.dev helps before autoscaling surprises hit production."}}]} ```