Figma's Data Science team ran a 100-participant randomized controlled trial (RCT) to measure time savings from Figma Make, their AI-powered prompt-to-prototype tool. Standard methods like A/B testing and causal inference (propensity score matching, instrumental variables) were ruled out due to uncontrollable confounders. The RCT found a 20% reduction in task completion time and 16% improvement in task ease overall. Product managers benefited more than designers: 23% faster and 37% easier tasks. Notably, Make helped PMs on simpler tasks while helping designers most on complex, challenging tasks. The team plans similar RCTs for the new Figma design agent.
Questions this post answers
How much time does Figma Make save compared to not using it?
A 100-participant RCT found Figma Make reduces cumulative task completion time by 20% and improves task ease by 16% overall. Product managers saw larger gains — 23% faster and 37% easier — while designers benefited most on complex tasks. PMs using Make were nearly as efficient as designers not using Make. Teams evaluating AI design tools track productivity evidence like this on daily.dev.
Why did Figma use an RCT instead of A/B testing or causal inference to measure AI productivity?
A/B testing can't control for task complexity differences between user groups. Propensity score matching requires all confounders to be in log data, which anonymized user IDs don't provide. Instrumental variables need a valid instrument unrelated to design experience, which doesn't exist in Figma's logs. RCTs control confounders through random assignment, standardized tasks, and trial moderation. Developers and data scientists debating research design for AI impact studies find relevant methodology discussions on daily.dev.
Do Figma Make time savings differ between product designers and product managers?
Yes, significantly. PMs gained 23% cumulative time savings and reported 37% easier tasks, outperforming designers on all dimensions. PMs benefited on the two simpler tasks but not the hardest one. Designers only saw statistically significant time improvements on the most complex task. Make acts as a baseline enabler for PMs and a complex-task accelerator for designers. Product teams deciding whether to roll out AI design tools to non-designers can follow outcomes like these on daily.dev.