--- title: Scaling StreamHub: Transitioning from Kinesis to Kafka... description: Atlassian's engineering team shares how they scaled StreamHub from 22 billion to 150 billion daily events by migrating from Amazon Kinesis to Apache Kafka via... canonical: https://daily.dev/posts/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events-mjsieq3rp twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events | daily.dev og:description: Atlassian's engineering team shares how they scaled StreamHub from 22 billion to 150 billion daily events by migrating from Amazon Kinesis to Apache Kafka via... og:url: https://daily.dev/posts/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events-mjsieq3rp og:image: https://api.daily.dev/og/posts/MJsIEQ3rP.png og:image:alt: Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events **[Atlassian](https://daily.dev/sources/atlassian)** · 15 min read · 1 upvotes · 0 comments ## Summary Atlassian's engineering team shares how they scaled StreamHub from 22 billion to 150 billion daily events by migrating from Amazon Kinesis to Apache Kafka via AWS MSK. The post details why Kinesis hit its limits (shard costs, retention constraints, consumer throughput), how Kafka Tiered Storage with S3 cut infrastructure costs, and the six critical operational challenges encountered at scale — including broker network saturation, S3 request rate storms during retention changes, managed control plane unavailability during AZ failures, and slow scaling operations during incidents. Solutions include intentional over-provisioning, ingress rate limiting, Kafka client quotas, cluster sharding with failover runbooks, and companion regions for compliant regional failover. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why would a company migrate from Amazon Kinesis to Kafka for high-volume event streaming? Kinesis becomes cost-prohibitive and operationally limiting at very high scale: its shard-based model ties cost linearly to throughput, extended retention beyond 24 hours requires expensive add-ons, and Enhanced Fan Out for low-latency multi-consumer delivery carries a steep premium. Kafka, via AWS MSK, supports cost-effective multi-day retention and scales consumer groups without proportional cost increases. _Teams weighing Kinesis versus Kafka for streaming at scale can compare real migration trade-offs on daily.dev._ ### What is Kafka Tiered Storage and how does it reduce infrastructure costs? Kafka Tiered Storage splits topic data into a local tier on broker EBS volumes for the most recent data (as little as 5 minutes) and a remote tier that asynchronously offloads closed log segments to low-cost object storage like Amazon S3, with retention configurable to multiple days. This avoids over-provisioning expensive EBS volumes for worst-case retention while historical reads come from S3, adding only a few hundred milliseconds of latency for non-latency-sensitive consumers. _Engineers evaluating storage cost strategies for Kafka can dig into tiered storage trade-offs via daily.dev._ ### Why did increasing then reverting Kafka tiered storage retention cause broker disk usage to spike? Temporarily raising tiered storage retention from 7 days to 21 days and then reverting back to 7 days triggered a large wave of delete requests against S3 for the older remote segments. That delete storm competed with the ongoing steady-state remote copy writes, causing remote writes to slow down, which in turn caused local broker disk utilization to climb quickly since data could no longer offload at the expected rate. _Operators tuning Kafka retention settings under load can learn from this incident pattern on daily.dev._ ## Similar posts on daily.dev - [AWS Kinesis vs Kafka: Choosing the Right Streaming Platform](https://daily.dev/posts/aws-kinesis-vs-kafka-choosing-the-right-streaming-platform-fhlmf8lir) · BigData Boutique blog · 6 upvotes · 0 comments - [How Reddit Migrated Petabyte-Scale Kafka from EC2 to Kubernetes](https://daily.dev/posts/how-reddit-migrated-petabyte-scale-kafka-from-ec2-to-kubernetes-wizskjogz) · ByteByteGo · 10 upvotes · 0 comments - [Scaling Jira cloud Migrations, One Bottleneck at a Time](https://daily.dev/posts/scaling-jira-cloud-migrations-one-bottleneck-at-a-time-w6sgyuoiq) · Atlassian · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#apache-kafka](https://daily.dev/tags/apache-kafka) [View this post on daily.dev](https://daily.dev/posts/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events-mjsieq3rp) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events","url":"https://daily.dev/posts/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events-mjsieq3rp","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events-mjsieq3rp"},"datePublished":"2026-07-28T17:48:40.963Z","dateModified":"2026-09-14T08:30:35.028Z","description":"Atlassian's engineering team shares how they scaled StreamHub from 22 billion to 150 billion daily events by migrating from Amazon Kinesis to Apache Kafka via...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/405b423f85edb92c31219bfe6173f353?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/405b423f85edb92c31219bfe6173f353?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Atlassian","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Atlassian","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/44750d8e66de4411bb477159b321bb43","url":"https://daily.dev/sources/atlassian"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events-mjsieq3rp","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,distributed-systems,apache-kafka","timeRequired":"PT15M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Atlassian","item":"https://daily.dev/sources/atlassian"},{"@type":"ListItem","position":3,"name":"Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events-mjsieq3rp#faq","mainEntity":[{"@type":"Question","name":"Why would a company migrate from Amazon Kinesis to Kafka for high-volume event streaming?","acceptedAnswer":{"@type":"Answer","text":"Kinesis becomes cost-prohibitive and operationally limiting at very high scale: its shard-based model ties cost linearly to throughput, extended retention beyond 24 hours requires expensive add-ons, and Enhanced Fan Out for low-latency multi-consumer delivery carries a steep premium. Kafka, via AWS MSK, supports cost-effective multi-day retention and scales consumer groups without proportional cost increases. Teams weighing Kinesis versus Kafka for streaming at scale can compare real migration trade-offs on daily.dev."}},{"@type":"Question","name":"What is Kafka Tiered Storage and how does it reduce infrastructure costs?","acceptedAnswer":{"@type":"Answer","text":"Kafka Tiered Storage splits topic data into a local tier on broker EBS volumes for the most recent data (as little as 5 minutes) and a remote tier that asynchronously offloads closed log segments to low-cost object storage like Amazon S3, with retention configurable to multiple days. This avoids over-provisioning expensive EBS volumes for worst-case retention while historical reads come from S3, adding only a few hundred milliseconds of latency for non-latency-sensitive consumers. Engineers evaluating storage cost strategies for Kafka can dig into tiered storage trade-offs via daily.dev."}},{"@type":"Question","name":"Why did increasing then reverting Kafka tiered storage retention cause broker disk usage to spike?","acceptedAnswer":{"@type":"Answer","text":"Temporarily raising tiered storage retention from 7 days to 21 days and then reverting back to 7 days triggered a large wave of delete requests against S3 for the older remote segments. That delete storm competed with the ongoing steady-state remote copy writes, causing remote writes to slow down, which in turn caused local broker disk utilization to climb quickly since data could no longer offload at the expected rate. Operators tuning Kafka retention settings under load can learn from this incident pattern on daily.dev."}}]} ```