Netflix engineers share a detailed account of building a real-time service dependency map at scale. The system combines eBPF network flows, IPC metrics, and distributed tracing into separate graph layers. A three-stage distributed aggregation pipeline resolves network intermediaries (load balancers, NAT gateways) into clean application-level edges using consistent hashing and Server-Sent Events for inter-stage communication. Key production challenges included Kafka consumer lag, hot nodes receiving 100x normal traffic, JVM garbage collection pressure from immutable data structures, and reactive streams complexity with Pekko. Solutions included splitting two-stage into three-stage processing, switching from gRPC to SSE, using mutable aggregators on the hot path, and tuning Kafka fetch parameters. A time-travel feature enables historical topology queries using time-windowed aggregator snapshots combined with property-level mutation tracking. The post emphasizes measurement-driven optimization and willingness to break conventional wisdom at extreme scale.
