Datadog engineers share six hard-won lessons from five years of running eBPF-powered workload protection at scale. The post covers: navigating kernel version and distribution compatibility pitfalls (verifier changes, hook point availability, function inlining); safely capturing kernel data (CO-RE offsets, TOCTOU races, paged-out user memory, non-linear skbs); maintaining consistent eBPF maps and user-space caches (LRU eviction quirks, blocking syscall abuse, event reordering); auditing eBPF as an attack surface (rootkit potential, helper abuse, tampering detection); handling conflicts with other eBPF tools sharing kernel resources (a real Cilium TC classifier incident); and measuring both user-space agent overhead and hidden kernel instrumentation cost. Each lesson includes concrete mitigations such as CI kernel matrices, the open-source ebpf-manager library, aggressive in-kernel event filtering, and dynamic re-hooking on module reload.
