A Microsoft Build session exploring how AI is transforming the data engineering workflow using Microsoft Fabric, VS Code, and GitHub Copilot. Two presenters demonstrate practical AI-native workflows: building a streaming ingestion pipeline from Kafka/EventStream into a medallion architecture (bronze/silver/gold Delta Lake tables) using Copilot in agent mode, and a local development setup where production OneLake data is mounted via Blobfuse into a VS Code dev container running local Spark. Key techniques include authoring AI-friendly codebases with consistent interfaces (e.g., a DataTransformer trait in Scala), using bounded context 'skills' to constrain Copilot's behavior, building deterministic CLI tools for AI to invoke, and parallelizing work across multiple Copilot CLI instances via GitHub Actions. The 'Ralph loop' concept is introduced as a mental model for giving AI a well-defined task within a bounded context.

31m watch time
130 Impressions