Yelp's data platform team built a system to visualize S3 partition access patterns by plotting partition key values against access event timestamps. These visualizations reveal distinct usage signatures — daily batch jobs, backfills, and ad hoc queries — attributed to specific IAM roles. Armed with granular usage data, teams confidently adopted more cost-effective S3 Storage Classes (leveraging Intelligent Tiering's 40–81% cost reductions), expanded deletion-based retention policies, and introduced a 'Default Access Retention' mechanism that gates cold data behind IAM policies requiring explicit approval. The combined effort reduced Yelp's petabyte-scale data lake S3 storage cost by 33%. The implementation uses Amazon S3 server access logs aggregated via SQL, and the same usage data also guided prioritization of Apache Iceberg migration efforts toward the most actively used tables.
