--- title: How Netflix Scaled Its Real-Time Service Map | daily.dev description: Netflix has detailed the redesign of the streaming pipeline behind Service Topology, its real-time service dependency map. The system processes network flow... canonical: https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How Netflix Scaled Its Real-Time Service Map | daily.dev og:description: Netflix has detailed the redesign of the streaming pipeline behind Service Topology, its real-time service dependency map. The system processes network flow... og:url: https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw og:image: https://api.daily.dev/og/posts/U7xT8LPtw.png og:image:alt: How Netflix Scaled Its Real-Time Service Map og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How Netflix Scaled Its Real-Time Service Map **[InfoQ](https://daily.dev/sources/infoq)** · 3 min read · 2 upvotes · 0 comments ## Summary Netflix has detailed the redesign of the streaming pipeline behind Service Topology, its real-time service dependency map. The system processes network flow data in three stages: initial aggregation from multi-region Kafka streams, intermediary resolution to convert raw network hops into logical application-to-application edges, and enrichment with metadata before graph persistence. A key fix addressed hot instances caused by popular destinations receiving up to 100x typical traffic. Apache Pekko Streams manages backpressure by pausing Kafka consumers rather than dropping records, trading freshness for completeness under load. Netflix also replaced gRPC between internal pipeline stages with server-sent events (SSE) for lighter serialization and better reactive backpressure compatibility. Consistent hashing distributes aggregator ownership across a dynamic fleet, automatically rebalancing when instances join or leave. The system also supports historical reconstruction using time-windowed snapshots and property-level mutation history, enabling engineers to examine dependency states around past incidents. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did Netflix replace gRPC with server-sent events for its internal Service Topology pipeline? Netflix found that gRPC's serialization overhead, connection-pool management, and streaming-response memory pressure became too expensive at its data volume between pipeline stages. Server-sent events proved lighter-weight and compatible with reactive backpressure. This internal transport change is separate from the gRPC API still exposed to Service Topology's external clients. _Weighing gRPC against lighter transport options for high-volume internal services? daily.dev surfaces real production tradeoffs like this._ ### How does Netflix's Service Topology pipeline avoid dropping data when graph storage falls behind? Instead of dropping records, Netflix propagates backpressure upstream through its processing stages using Apache Pekko Streams until the Kafka consumer pauses, leaving records buffered in Kafka until capacity returns. This produces delayed freshness during load spikes rather than data loss or an incomplete dependency map, which Netflix considers preferable to stale batch-generated maps during incidents. _Designing backpressure-aware streaming pipelines? daily.dev tracks architecture patterns like this for distributed systems engineers._ ### How does Netflix distribute aggregator ownership across its Service Topology processing fleet? Each processing instance reads the same current list of healthy instances from the service registry and applies consistent hashing to determine which instance owns each aggregator. When an instance joins or leaves the fleet, only the affected aggregators automatically move to new owners, avoiding a separate rebalancing process entirely. _Scaling stateful processing fleets with consistent hashing? daily.dev follows production patterns like Netflix's approach._ ## Similar posts on daily.dev - [How Netflix Maps Thousands of Microservices in Real-Time](https://daily.dev/posts/how-netflix-maps-thousands-of-microservices-in-real-time-btv6gxgki) · InfoQ · 0 upvotes · 0 comments - [Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned](https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7) · Netflix TechBlog · 26 upvotes · 0 comments - [From Silos to Service Topology: Why Netflix Built a Real-Time Service Map](https://daily.dev/posts/from-silos-to-service-topology-why-netflix-built-a-real-time-service-map-rlefmifnv) · Netflix TechBlog · 20 upvotes · 2 comments - [How Netflix Built a Real-Time Distributed Graph for Internet Scale](https://daily.dev/posts/how-netflix-built-a-real-time-distributed-graph-for-internet-scale-1xkysklft) · ByteByteGo · 88 upvotes · 0 comments --- Tags: [#linux](https://daily.dev/tags/linux), [#microservices](https://daily.dev/tags/microservices), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#apache-kafka](https://daily.dev/tags/apache-kafka), [#server-sent-events](https://daily.dev/tags/server-sent-events) [View this post on daily.dev](https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How Netflix Scaled Its Real-Time Service Map","url":"https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw"},"datePublished":"2026-08-11T12:24:01.442Z","dateModified":"2026-09-14T07:18:09.398Z","description":"Netflix has detailed the redesign of the streaming pipeline behind Service Topology, its real-time service dependency map. The system processes network flow...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d5dd60454188e72351def48f12e219b1?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d5dd60454188e72351def48f12e219b1?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"InfoQ","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"InfoQ","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/afc3bced3e1e4b188dd9127017a60e0c","url":"https://daily.dev/sources/infoq"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"linux,microservices,distributed-systems,apache-kafka,server-sent-events","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"InfoQ","item":"https://daily.dev/sources/infoq"},{"@type":"ListItem","position":3,"name":"How Netflix Scaled Its Real-Time Service Map"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw#faq","mainEntity":[{"@type":"Question","name":"Why did Netflix replace gRPC with server-sent events for its internal Service Topology pipeline?","acceptedAnswer":{"@type":"Answer","text":"Netflix found that gRPC's serialization overhead, connection-pool management, and streaming-response memory pressure became too expensive at its data volume between pipeline stages. Server-sent events proved lighter-weight and compatible with reactive backpressure. This internal transport change is separate from the gRPC API still exposed to Service Topology's external clients. Weighing gRPC against lighter transport options for high-volume internal services? daily.dev surfaces real production tradeoffs like this."}},{"@type":"Question","name":"How does Netflix's Service Topology pipeline avoid dropping data when graph storage falls behind?","acceptedAnswer":{"@type":"Answer","text":"Instead of dropping records, Netflix propagates backpressure upstream through its processing stages using Apache Pekko Streams until the Kafka consumer pauses, leaving records buffered in Kafka until capacity returns. This produces delayed freshness during load spikes rather than data loss or an incomplete dependency map, which Netflix considers preferable to stale batch-generated maps during incidents. Designing backpressure-aware streaming pipelines? daily.dev tracks architecture patterns like this for distributed systems engineers."}},{"@type":"Question","name":"How does Netflix distribute aggregator ownership across its Service Topology processing fleet?","acceptedAnswer":{"@type":"Answer","text":"Each processing instance reads the same current list of healthy instances from the service registry and applies consistent hashing to determine which instance owns each aggregator. When an instance joins or leaves the fleet, only the affected aggregators automatically move to new owners, avoiding a separate rebalancing process entirely. Scaling stateful processing fleets with consistent hashing? daily.dev follows production patterns like Netflix's approach."}}]} ```