Elastic Security Labs describes a benchmarking framework for evaluating LLMs in agentic Security Operations Center (SOC) workflows. Unlike generic leaderboards that score text quality, this framework seeds a realistic synthetic intrusion (the Chrysalis backdoor campaign) into a live Elastic deployment, runs every model through the same agent with identical tools and prompts, and captures full execution traces — every tool call, parameter, and result. Three evaluation suites are covered: an agent suite testing seven capability categories (alert triage, entity analytics, threat hunting, detection rule authoring, workflow authoring, workflow triggering, and multi-step response); Attack Discovery, which grades how well models correlate raw alerts into coherent attack narratives; and Automatic Migration, which measures how faithfully models port Splunk detection rules into Elastic. Scoring is blind (models anonymized before judging), caps unsubstantiated answers at 6/10, and separates reliability failures from quality scores. The key finding: model rankings flip depending on the capability, making per-task evaluation essential for production SOC deployments.
