Building May Be Cheap Now. Being Wrong Still Isn't.
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
AI has drastically reduced the cost of building software but has done nothing to reduce the cost of building the wrong thing. Drawing on Robbie Allen's distinction between build costs and ownership costs, and Ronny Kohavi's Bing example (100 engineers, a year of sunk time, a full rollback of a feature that failed experiments twice), the piece argues that discovery research is now more valuable than ever because it's cheap relative to the risk of shipping wrong. The proposed rule: do discovery research when uncertain about the problem, do prototyping and experiments when uncertain about the solution. A client case is described where a request to add an AI assistant was reframed after research revealed users struggled with evaluating relevance, not finding content, leading to a different, cheaper fix.
Questions this post answers
What happened when Bing shipped a third search pane feature despite failed experiments?
Roughly 100 engineers built a third pane for Bing's search window, and the A/B experiments failed to show value. It still shipped to all users based on a strategic business rationale. A year later, after further experiments again failed to show value, the feature was rolled back at significant organizational cost, illustrating that a shipped product carries costs a discarded prototype never does. Teams weighing whether to ship an unproven feature can find similar product decision case studies on daily.dev.
How do I decide whether my team needs discovery research or just prototyping and testing?
Do discovery research when uncertain about the problem itself, and do prototyping and user testing when uncertain about the solution. For example, a team that already knows users abandon checkout and just needs to test a progress indicator versus guest checkout doesn't need a discovery study; building and testing both options teaches more in two weeks than research would. daily.dev helps product-minded developers weigh research versus rapid prototyping trade-offs like this one.
2K Impressions1 Comment