Datadog's engineering team describes how they scaled BewAIre, their LLM-based malicious code detection system, from pull request analysis to full software supply chain scanning. The system uses a two-stage pipeline: a cheap filter model that screens all changes, escalating only suspicious ones to an agentic investigation loop with tool access (GitHub APIs, osv.dev, SCA). For scanning full dependency packages, they tested three strategies and found that providing the agent a codemap plus a ReadFile tool yielded the best accuracy, latency, and cost. They also built a crawler that monitors npm and PyPI registries every 10 minutes, detecting the node-ipc hack within 5 minutes. The updated architecture achieves a 95.5% detection rate on confirmed malicious packages with low false positives, and improved overall accuracy from 97.4% to 99.86% on pull requests by eliminating false positives.
