A step-by-step guide to building a production-grade document ingestion pipeline for RAG systems using Apache Airflow, FastAPI, and PostgreSQL. Covers the full architecture: a FastAPI service that accepts PDF uploads with SHA-256 deduplication, a PostgreSQL schema with documents, chunks, and pipeline_runs tables, and a 5-task Airflow DAG (fetch, parse, chunk, validate, mark complete). Key design principles include idempotency via content hashing, status tracking, separation of ingestion from processing, and granular per-document error handling. Data is passed between tasks via JSON files rather than XCom to avoid database size limits. The setup uses Docker Compose with a shared volume so Airflow and the API service can access uploaded files without network transfers.
