Materialize shares how they built an LLM-based bug-finding system using Claude (Opus 4.6/4.7) that scans pull requests, commits, and source files. The system runs automated coding agent sessions with carefully crafted prompts that categorize bugs by severity, cross-check against existing issues, and use tools like Trailmark and LSP for efficient codebase traversal. Key lessons include: always manually verifying LLM-reported bugs, using existing test frameworks as oracles, applying for safeguard adjustments with Anthropic/OpenAI to avoid policy blocks, and iterating on prompts to reduce false positives. The approach has uncovered hundreds of bugs not caught by existing test suites, though it complements rather than replaces systematic testing.
