MDPI, an open access scientific publisher, describes how its Data Engineering and AI teams use Polars to process about 800GB of Parquet files daily, much of it textual data for NLP and language model inference. The team chose Polars for its speed, Rust-based performance combined with a Python-friendly API, and open-source alignment with an on-premise, non-cloud infrastructure. Custom Rust-based Polars plugins enable efficient text cleaning and language detection, while integration with Apache Arrow, predicate pushdown, ADBC drivers, and on-premise MinIO S3 storage supports a columnar, high-throughput pipeline orchestrated with Apache Airflow.
Table of contents
IntroductionAbout MDPIMDPI AIWhy we chose PolarsWhat made the differenceLearn more about MDPIQuestions this post answers
Why would a company choose Polars over pandas for large-scale data processing?
Polars combines the speed of an optimized Rust library with the ease of a Python environment, offering LazyFrame execution, predicate pushdown, and tight integration with Apache Arrow for efficient batch processing. MDPI uses it to process about 800GB of Parquet files daily on-premise, including NLP transformations and language model inference, without needing costly cloud infrastructure. Developers comparing Polars and pandas for large datasets can track real adoption stories like this on daily.dev.
Can you write custom data processing plugins for Polars in Rust?
Yes, Polars supports custom plugins written in Rust, which allows teams to build optimized functions such as text cleaning or language detection that run efficiently in batches. This level of customization goes beyond what typical processing engines or pure SQL-based data engineering can offer, and is used in production for NLP workloads on large text datasets. Teams building custom NLP pipelines can follow practical Polars plugin patterns like this on daily.dev.