--- title: "Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned" url: https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7 source_url: https://netflixtechblog.com/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-f4b792f3f0d8 type: article source: "Netflix TechBlog" published: 2026-07-14T07:22:59.815Z updated: 2026-07-14T14:21:19.072Z tags: ["distributed-systems", "apache-kafka", "scala"] reading_time: 26 upvotes: 26 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned **[Netflix TechBlog](https://daily.dev/sources/netflix)** · 26 min read · 26 upvotes · 0 comments ## Summary Netflix engineers share a detailed account of building a real-time service dependency map at scale. The system combines eBPF network flows, IPC metrics, and distributed tracing into separate graph layers. A three-stage distributed aggregation pipeline resolves network intermediaries (load balancers, NAT gateways) into clean application-level edges using consistent hashing and Server-Sent Events for inter-stage communication. Key production challenges included Kafka consumer lag, hot nodes receiving 100x normal traffic, JVM garbage collection pressure from immutable data structures, and reactive streams complexity with Pekko. Solutions included splitting two-stage into three-stage processing, switching from gRPC to SSE, using mutable aggregators on the hot path, and tuning Kafka fetch parameters. A time-travel feature enables historical topology queries using time-windowed aggregator snapshots combined with property-level mutation tracking. The post emphasizes measurement-driven optimization and willingness to break conventional wisdom at extreme scale. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [How Netflix Maps Thousands of Microservices in Real-Time](https://daily.dev/posts/how-netflix-maps-thousands-of-microservices-in-real-time-btv6gxgki) · InfoQ · 0 upvotes · 0 comments - [From Silos to Service Topology: Why Netflix Built a Real-Time Service Map](https://daily.dev/posts/from-silos-to-service-topology-why-netflix-built-a-real-time-service-map-rlefmifnv) · Netflix TechBlog · 20 upvotes · 2 comments - [How Netflix Scaled Its Real-Time Service Map](https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw) · InfoQ · 2 upvotes · 0 comments --- Tags: [#distributed-systems](https://daily.dev/tags/distributed-systems), [#apache-kafka](https://daily.dev/tags/apache-kafka), [#scala](https://daily.dev/tags/scala) [View this post on daily.dev](https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7)