Lessons learned from running a large gRPC mesh at Datadog, including challenges of implementing a networking solution for a complex application, how gRPC facilitates communication within Kubernetes clusters, and best practices for running gRPC services in a high-scale Kubernetes environment.
Table of contents
Use Kubernetes headless services and gRPC round-robin to avoid load balance issuesLeverage TLS to handle IP recyclingSet MAX_CONNECTION_AGE to force gRPC to re-resolve from DNS, ensuring proper scale-out detectionSet the keepalive channel option to mitigate the effects of silent connection dropsProperly monitor your servicesAcknowledgments93 Impressions