A podcast episode features Suneet Malhotra, Senior Manager of Test Engineering at Motorola Solutions, discussing his five-agent agentic SDLC pipeline built with MCPs. Topics include using Cohen's kappa to evaluate multiple LLMs acting as judges, and shifting QA left by adding a specification enrichment stage right after the design phase. Links to companion code, research papers on cross-layer observability for LLM-assisted test automation, and Suneet's professional profiles are provided.
Questions this post answers
What is Cohen's kappa used for when evaluating multiple LLMs as judges?
Cohen's kappa measures agreement between multiple large language models acting as judges in an evaluation pipeline, going beyond simple accuracy to account for agreement that could occur by chance. Suneet Malhotra, Senior Manager of Test Engineering at Motorola Solutions, applies this statistic within a five-agent agentic SDLC pipeline built with MCPs to validate LLM-based judgments more rigorously. Teams building LLM-as-judge evaluation pipelines can follow emerging QA methodology like this via daily.dev.
What does shifting QA left through specification enrichment mean in an SDLC?
It means adding a dedicated specification enrichment stage immediately after the design phase, so requirements are clarified and strengthened before development begins rather than catching gaps during later testing. This approach, part of a five-agent agentic SDLC pipeline described by a Motorola Solutions test engineering manager, aims to improve requirement quality earlier in the lifecycle. Engineers redesigning their QA process around earlier requirement checks can track similar practices on daily.dev.