Meta's Analytics team built DEmate, an AI assistant tailored to their proprietary data engineering stack. General LLMs failed because they lacked knowledge of Meta's internal tools, custom SQL macros, and UDFs. The solution was a 'Recipe Architecture' — a four-phase execution loop (auto-selection, context injection, code generation, validation) backed by ~70 DE-specific recipes. Recipe Chaining with Progressive Disclosure keeps LLM context focused and reduces hallucinations. An AI-powered code review system enforces 20 quality rules for community-contributed recipes. Asynchronous validation was solved by integrating an Analytics Agent that monitors long-running test jobs and auto-updates diffs. DEmate is deployed across Meta's IDE extension, SQL interface, and task tracking tools, and an Omni Plugin architecture allows model-agnostic future-proofing. In five months, DEmate reached 3,500 weekly active users with an 80% code acceptance rate.

8m read timeFrom medium.com
Post cover image
Table of contents
The Context Gap: Why General AI Struggled at Meta’s Data Engineering workflowsThe Breakthrough: Recipe ArchitectureEfficiency at Scale: Recipe ChainingThe Sous-Chef: AI-Powered Code ReviewsGet Analytics at Meta’s stories in your inboxSimmering in the Background: Asynchronous ValidationA Seamless, Multi-Surface EcosystemFuture-Proofing: The Omni Plugin and Agent OrchestrationClosing: The Future of Intent-Based Engineering
186 Impressions