A Salesforce engineering architect describes how Security Center evolved from a conversational AI interface into a stateful, Agentforce-powered investigations platform for security incident response. The interview covers challenges in validating non-deterministic LLM behavior at scale (solved via AI-driven evaluation pipelines that boosted testing throughput 10-20X), managing finite context windows across large telemetry datasets, mitigating hallucinations in security reasoning, and architecting extensible data models across heterogeneous telemetry systems. It also touches on ongoing work to ground responses in Salesforce-specific security knowledge rather than relying solely on public foundation models.
Questions this post answers
How do you test AI agents that give different responses to the same prompt every time?
Non-deterministic LLM behavior can be validated using AI-driven evaluation pipelines where one AI system checks the outputs of another. Simulated customer interactions are fed into the agent, and LLM evaluators judge whether responses match the intended investigative outcome rather than checking for exact wording, increasing testing throughput by roughly 10-20X compared to manual validation. daily.dev surfaces engineering approaches like AI-evaluating-AI for teams building trustworthy agent workflows.
How do you handle large security telemetry datasets that exceed an LLM's context window during an investigation?
Large telemetry data is partitioned into smaller contextual segments and compressed using AI-driven summarization pipelines before being fed into the agent, avoiding raw data overload of the context window. This approach was built to support long-running security investigations spanning multiple telemetry systems where reasoning context could otherwise collapse mid-investigation. engineers tackling context-window limits in agent design track these patterns on daily.dev.