Pinterest engineers built MIQPS (Minimal Important Query Param Set), a data-driven URL normalization system that replaces manual allowlists and denylists for deduplication across millions of merchant domains. Instead of relying on canonical tags or hand-crafted rules, MIQPS renders pages and generates content fingerprints to determine whether removing a query parameter changes the underlying page content. Parameters that cause content changes above a threshold are retained; others are stripped. The system separates expensive offline analysis from lightweight runtime processing, applies anomaly detection to prevent regressions, and uses early-exit logic to minimize unnecessary page renders. This approach significantly reduces duplicate fetch, render, and indexing costs in Pinterest's large-scale content ingestion pipeline.
