Uber automated the rotation of over 100,000 Kerberos keytabs across their infrastructure by building a custom system that integrates with their Secret Management Platform. The solution addresses key challenges including scale complexity and service disruption risks through failure-domain-aware automation, rate limiting, and careful timing coordination. The system minimizes authentication failures during rotation by ensuring sufficient TGT validity time and maintaining both old and new key versions on servers during transitions. Safety measures include blast radius minimization, cluster-based allowlists for gradual rollout, and automatic deletion capabilities. Uber is now exploring PKINIT as the next evolution to replace keytabs with certificate-based authentication.
