Apache Parquet is a columnar binary format optimized for analytics workloads like Spark and Athena. Redpanda Connect can encode streaming data directly into compressed Parquet files and write them to S3. The tutorial demonstrates configuring a pipeline that reads from a Redpanda topic, batches messages, encodes them using a defined schema with the parquet_encode processor, applies zstd compression, and outputs to S3. This approach enables a single data stream to serve both real-time applications (JSON) and analytics systems (Parquet).
Table of contents
Streaming data to S3 for analytics with Parquet #Example Redpanda Connect pipeline #Cleanup and best practices #What’s next #4 Impressions