---
title: "From Archival to Access: Config-Driven Data Pipelines"
url: https://daily.dev/posts/from-archival-to-access-config-driven-data-pipelines-jfnhe1zgb
source_url: https://www.uber.com/blog/from-archival-to-access/
type: article
source: "Uber Engineering"
published: 2025-06-05T13:32:39.475Z
updated: 2025-06-05T13:33:02.940Z
tags: ["backend", "compliance", "data-engineering", "apache-spark", "apache-hadoop"]
reading_time: 13
upvotes: 1
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# From Archival to Access: Config-Driven Data Pipelines

**[Uber Engineering](https://daily.dev/sources/uber)** · 13 min read · 1 upvotes · 0 comments

## Summary

Uber's Compliance Data Store team developed a config-driven archival and retrieval framework to manage terabytes of regulatory data efficiently. The system automatically moves data from hot storage (HDFS) to cold storage (S3) based on configurable policies, while providing on-demand retrieval capabilities. The solution reduced manual intervention by 90% and successfully handles over 500 regulatory reports, addressing challenges like schema evolution, data consistency during backfills, and resource optimization. The framework uses MySQL for metadata management, Apache Airflow for orchestration, and includes a user-friendly interface for self-service data retrieval.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.uber.com/blog/from-archival-to-access/>

## Similar posts on daily.dev

- [Don’t just attend KubeCon \+ CloudNativeCon, Merge Forward your experience\!](https://daily.dev/posts/don-t-just-attend-kubecon-cloudnativecon-merge-forward-your-experience--l0rpp73x8) · CNCF · 0 upvotes · 0 comments
- [Announcing H2 2026 KCDs](https://daily.dev/posts/announcing-h2-2026-kcds-m96goajm1) · CNCF · 1 upvotes · 0 comments
- [Two months of Open Community Groups](https://daily.dev/posts/two-months-of-open-community-groups-asf52zhbs) · CNCF · 0 upvotes · 0 comments

---

Tags: [#backend](https://daily.dev/tags/backend), [#compliance](https://daily.dev/tags/compliance), [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#apache-hadoop](https://daily.dev/tags/apache-hadoop)

[View this post on daily.dev](https://daily.dev/posts/from-archival-to-access-config-driven-data-pipelines-jfnhe1zgb)