Pinterest's ML team shares a detailed investigation into why new L1 conversion (CVR) models showed strong offline improvements (20–45% LogMAE reduction) but neutral or negative online A/B results. The root causes were identified as: (1) feature coverage gaps where high-impact features existed in training logs but were never onboarded into the L1 embedding serving path, and (2) embedding version skew in two-tower architectures where query and Pin towers ran on different model checkpoints in production. Beyond these, funnel alignment issues (recall ceilings) and metric mismatch between offline loss metrics and online CPA further explained the gap. The team's key takeaway is that online-offline discrepancy should be treated as a design constraint from the start, not a post-hoc debugging problem.
