--- title: Building Service Topology at Scale: Architecture,... description: Netflix engineers share a detailed account of building a real-time service dependency map at scale. The system combines eBPF network flows, IPC metrics, and... canonical: https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned | daily.dev og:description: Netflix engineers share a detailed account of building a real-time service dependency map at scale. The system combines eBPF network flows, IPC metrics, and... og:url: https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7 og:image: https://api.daily.dev/og/posts/QpTWMjvF7.png og:image:alt: Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned **[Netflix TechBlog](https://daily.dev/sources/netflix)** · 26 min read · 26 upvotes · 0 comments ## Summary Netflix engineers share a detailed account of building a real-time service dependency map at scale. The system combines eBPF network flows, IPC metrics, and distributed tracing into separate graph layers. A three-stage distributed aggregation pipeline resolves network intermediaries (load balancers, NAT gateways) into clean application-level edges using consistent hashing and Server-Sent Events for inter-stage communication. Key production challenges included Kafka consumer lag, hot nodes receiving 100x normal traffic, JVM garbage collection pressure from immutable data structures, and reactive streams complexity with Pekko. Solutions included splitting two-stage into three-stage processing, switching from gRPC to SSE, using mutable aggregators on the hot path, and tuning Kafka fetch parameters. A time-travel feature enables historical topology queries using time-windowed aggregator snapshots combined with property-level mutation tracking. The post emphasizes measurement-driven optimization and willingness to break conventional wisdom at extreme scale. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did switching from gRPC to Server-Sent Events help with Netflix's service topology pipeline hot node problem? Server-Sent Events reduced resource consumption on both sender and receiver sides compared to gRPC, whose serialization overhead, connection pool management, and memory pressure for streaming responses consumed more CPU than business logic. SSE offered a lightweight HTTP-based protocol with minimal serialization and natural backpressure integration with reactive streams, making it better suited for streaming large volumes of pre-aggregated data than RPC-oriented gRPC. _daily.dev surfaces engineering trade-off writeups like this for teams choosing between gRPC and lighter streaming protocols._ ### Why did Netflix use three pipeline stages instead of two for network flow aggregation? A two-stage design (aggregate then resolve-and-persist) overwhelmed the resolution stage because collecting all flows for a popular intermediary on one instance created hot nodes doing both compute-heavy resolution and I/O-heavy enrichment simultaneously. Splitting into three stages isolates resolution (stage 2) from enrichment and persistence (stage 3), redistributing data twice via consistent hashing so no single instance handles 100x the typical load even for very popular services. _engineers designing multi-stage aggregation pipelines can track this kind of scaling lesson on daily.dev._ ### Why did switching from immutable to mutable data structures reduce garbage collection pauses in a Scala streaming pipeline? Mutable aggregators on the hot path reduced heap allocation by over 50% and cut GC pause time significantly, because immutable structures created a new object on every update, overwhelming the garbage collector when processing millions of records per second. The trade-off required more careful code review, and immutability was retained everywhere except the specific hot path where performance data justified the change. _developers weighing immutability against GC pressure at scale can follow this kind of trade-off analysis on daily.dev._ ## Similar posts on daily.dev - [How Netflix Maps Thousands of Microservices in Real-Time](https://daily.dev/posts/how-netflix-maps-thousands-of-microservices-in-real-time-btv6gxgki) · InfoQ · 0 upvotes · 0 comments - [From Silos to Service Topology: Why Netflix Built a Real-Time Service Map](https://daily.dev/posts/from-silos-to-service-topology-why-netflix-built-a-real-time-service-map-rlefmifnv) · Netflix TechBlog · 20 upvotes · 2 comments - [How Netflix Scaled Its Real-Time Service Map](https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw) · InfoQ · 2 upvotes · 0 comments - [How and Why Netflix Built a Real-Time Distributed Graph: Part 1 — Ingesting and Processing Data Streams at Internet Scale](https://daily.dev/posts/how-and-why-netflix-built-a-real-time-distributed-graph-part-1-ingesting-and-processing-data-stre-ho3v4j4yn) · Netflix TechBlog · 82 upvotes · 0 comments - [How Netflix Built a Real-Time Distributed Graph for Internet Scale](https://daily.dev/posts/how-netflix-built-a-real-time-distributed-graph-for-internet-scale-1xkysklft) · ByteByteGo · 88 upvotes · 0 comments --- Tags: [#distributed-systems](https://daily.dev/tags/distributed-systems), [#apache-kafka](https://daily.dev/tags/apache-kafka), [#scala](https://daily.dev/tags/scala) [View this post on daily.dev](https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned","url":"https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7"},"datePublished":"2026-07-14T07:22:59.815Z","dateModified":"2026-09-14T09:04:12.276Z","description":"Netflix engineers share a detailed account of building a real-time service dependency map at scale. The system combines eBPF network flows, IPC metrics, and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e582e465ad3b41328a78d21c1146a41d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e582e465ad3b41328a78d21c1146a41d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Netflix TechBlog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Netflix TechBlog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/netflix","url":"https://daily.dev/sources/netflix"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":26},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"distributed-systems,apache-kafka,scala","timeRequired":"PT26M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Netflix TechBlog","item":"https://daily.dev/sources/netflix"},{"@type":"ListItem","position":3,"name":"Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7#faq","mainEntity":[{"@type":"Question","name":"Why did switching from gRPC to Server-Sent Events help with Netflix's service topology pipeline hot node problem?","acceptedAnswer":{"@type":"Answer","text":"Server-Sent Events reduced resource consumption on both sender and receiver sides compared to gRPC, whose serialization overhead, connection pool management, and memory pressure for streaming responses consumed more CPU than business logic. SSE offered a lightweight HTTP-based protocol with minimal serialization and natural backpressure integration with reactive streams, making it better suited for streaming large volumes of pre-aggregated data than RPC-oriented gRPC. daily.dev surfaces engineering trade-off writeups like this for teams choosing between gRPC and lighter streaming protocols."}},{"@type":"Question","name":"Why did Netflix use three pipeline stages instead of two for network flow aggregation?","acceptedAnswer":{"@type":"Answer","text":"A two-stage design (aggregate then resolve-and-persist) overwhelmed the resolution stage because collecting all flows for a popular intermediary on one instance created hot nodes doing both compute-heavy resolution and I/O-heavy enrichment simultaneously. Splitting into three stages isolates resolution (stage 2) from enrichment and persistence (stage 3), redistributing data twice via consistent hashing so no single instance handles 100x the typical load even for very popular services. engineers designing multi-stage aggregation pipelines can track this kind of scaling lesson on daily.dev."}},{"@type":"Question","name":"Why did switching from immutable to mutable data structures reduce garbage collection pauses in a Scala streaming pipeline?","acceptedAnswer":{"@type":"Answer","text":"Mutable aggregators on the hot path reduced heap allocation by over 50% and cut GC pause time significantly, because immutable structures created a new object on every update, overwhelming the garbage collector when processing millions of records per second. The trade-off required more careful code review, and immutability was retained everywhere except the specific hot path where performance data justified the change. developers weighing immutability against GC pressure at scale can follow this kind of trade-off analysis on daily.dev."}}]} ```