--- title: Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79% description: An ML training job was exhausting worker memory during Snowflake data fetches, peaking at 88.8 GiB RSS before training even started. The root cause was the... canonical: https://daily.dev/posts/apache-arrow-cut-snowflake-fetch-memory-growth-by-up-to-79--0tujnnbzf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79% | daily.dev og:description: An ML training job was exhausting worker memory during Snowflake data fetches, peaking at 88.8 GiB RSS before training even started. The root cause was the... og:url: https://daily.dev/posts/apache-arrow-cut-snowflake-fetch-memory-growth-by-up-to-79--0tujnnbzf og:image: https://api.daily.dev/og/posts/0tuJNnBZf.png og:image:alt: Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79% og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79% **[Ramp Engineering](https://daily.dev/sources/ramp_eng)** · 5 min read · 5 upvotes · 0 comments ## Summary An ML training job was exhausting worker memory during Snowflake data fetches, peaking at 88.8 GiB RSS before training even started. The root cause was the shared Snowflake-to-pandas path materializing results as Python objects and SQLAlchemy rows before building the final DataFrame — consuming roughly 3x the final DataFrame size at peak. By adding an Apache Arrow path for supported reads, median peak RSS growth dropped 60–79% across three query shapes, and peak Python allocations fell 69–87%. In one large workflow, the lower fetch peak doubled the training-data window on the same cluster size. The implementation required normalizing dtype differences between Arrow and SQLAlchemy (integer widths, aggregate representations, identifier casing) and maintaining a SQLAlchemy fallback for unsupported cases. Because all ML jobs shared the same data-access layer, no query rewrites were needed. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much does using Apache Arrow instead of SQLAlchemy reduce memory usage when fetching Snowflake query results into pandas? Switching to an Arrow-based fetch path reduced median peak process RSS growth by 60-79% across three tested query shapes (a six-column projection over 100,000 rows, a 187-column read over 50,000 rows, and an aggregate query on a wide analytical table). Peak Python allocations measured by tracemalloc fell 69-87% over the same runs, since Arrow skips building intermediate Python objects and SQLAlchemy row representations before pandas assembles the final DataFrame. _Track memory-saving patterns like this Arrow fetch path alongside other data-pipeline changes on daily.dev._ ### What causes high memory usage when loading Snowflake query results into a pandas DataFrame via SQLAlchemy? The SQLAlchemy path materializes each result row as Python objects and SQLAlchemy row objects before pandas builds the final DataFrame, and these temporary representations can require roughly three times the final DataFrame size at peak. In one case, a cold fetch peaked at 88.8 GiB RSS versus 17.6 GiB when loading the same data already materialized as local Parquet. _Compare data-loading architectures like this before hitting a memory wall in production, curated on daily.dev._ ### What edge cases arise when replacing SQLAlchemy with Arrow for reading data from Snowflake into pandas? Arrow and SQLAlchemy can differ in integer widths, aggregate representations, and identifier casing, requiring normalization in a shared layer. Testing nine query shapes across four tables found eight matched after normalization, but a wide SELECT * with ten nullable columns differed: SQLAlchemy returned Python None in object columns while Arrow used typed float or datetime columns with NaN or NaT, so that shape stayed on the SQLAlchemy path. _Developers evaluating an Arrow migration can follow real compatibility gotchas like this on daily.dev._ ## Similar posts on daily.dev - [What Can We Do When Memory Becomes the New Bottleneck in Data Engineering?](https://daily.dev/posts/what-can-we-do-when-memory-becomes-the-new-bottleneck-in-data-engineering--h1dho6aid) · Towards Data Science · 0 upvotes · 0 comments - [Introducing Apache Arrow Support in mssql-python](https://daily.dev/posts/introducing-apache-arrow-support-in-mssql-python-yq9micn57) · Planet Python · 0 upvotes · 0 comments - [10X more data, same 4 seconds: single-query scaling in Redpanda SQL on 1TB](https://daily.dev/posts/10x-more-data-same-4-seconds-single-query-scaling-in-redpanda-sql-on-1tb-qj6rt0fu2) · Redpanda · 0 upvotes · 0 comments - [3.7x Faster Pipelines: Benchmarking Arrow & ADBC vs. SQLAlchemy for EL](https://daily.dev/posts/3-7x-faster-pipelines-benchmarking-arrow-adbc-vs-sqlalchemy-for-el-ntb759wmg) · dltHub · 1 upvotes · 0 comments - [Pandas Should Go Extinct](https://daily.dev/posts/pandas-should-go-extinct-1jqzeauiq) · Lobsters · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#snowflake](https://daily.dev/tags/snowflake), [#pandas](https://daily.dev/tags/pandas), [#apache-arrow](https://daily.dev/tags/apache-arrow) [View this post on daily.dev](https://daily.dev/posts/apache-arrow-cut-snowflake-fetch-memory-growth-by-up-to-79--0tujnnbzf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79%","url":"https://daily.dev/posts/apache-arrow-cut-snowflake-fetch-memory-growth-by-up-to-79--0tujnnbzf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/apache-arrow-cut-snowflake-fetch-memory-growth-by-up-to-79--0tujnnbzf"},"datePublished":"2026-07-22T21:07:08.291Z","dateModified":"2026-09-14T06:02:31.237Z","description":"An ML training job was exhausting worker memory during Snowflake data fetches, peaking at 88.8 GiB RSS before training even started. The root cause was the...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0a35f01a520a823373531c4e05dac383?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0a35f01a520a823373531c4e05dac383?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Ramp Engineering","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Ramp Engineering","logo":"https://media.daily.dev/image/upload/s--4x5H_pLU--/f_auto/v1704280953/logos/ramp.jpg","url":"https://daily.dev/sources/ramp_eng"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/apache-arrow-cut-snowflake-fetch-memory-growth-by-up-to-79--0tujnnbzf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,snowflake,pandas,apache-arrow","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Ramp Engineering","item":"https://daily.dev/sources/ramp_eng"},{"@type":"ListItem","position":3,"name":"Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79%"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/apache-arrow-cut-snowflake-fetch-memory-growth-by-up-to-79--0tujnnbzf#faq","mainEntity":[{"@type":"Question","name":"How much does using Apache Arrow instead of SQLAlchemy reduce memory usage when fetching Snowflake query results into pandas?","acceptedAnswer":{"@type":"Answer","text":"Switching to an Arrow-based fetch path reduced median peak process RSS growth by 60-79% across three tested query shapes (a six-column projection over 100,000 rows, a 187-column read over 50,000 rows, and an aggregate query on a wide analytical table). Peak Python allocations measured by tracemalloc fell 69-87% over the same runs, since Arrow skips building intermediate Python objects and SQLAlchemy row representations before pandas assembles the final DataFrame. Track memory-saving patterns like this Arrow fetch path alongside other data-pipeline changes on daily.dev."}},{"@type":"Question","name":"What causes high memory usage when loading Snowflake query results into a pandas DataFrame via SQLAlchemy?","acceptedAnswer":{"@type":"Answer","text":"The SQLAlchemy path materializes each result row as Python objects and SQLAlchemy row objects before pandas builds the final DataFrame, and these temporary representations can require roughly three times the final DataFrame size at peak. In one case, a cold fetch peaked at 88.8 GiB RSS versus 17.6 GiB when loading the same data already materialized as local Parquet. Compare data-loading architectures like this before hitting a memory wall in production, curated on daily.dev."}},{"@type":"Question","name":"What edge cases arise when replacing SQLAlchemy with Arrow for reading data from Snowflake into pandas?","acceptedAnswer":{"@type":"Answer","text":"Arrow and SQLAlchemy can differ in integer widths, aggregate representations, and identifier casing, requiring normalization in a shared layer. Testing nine query shapes across four tables found eight matched after normalization, but a wide SELECT * with ten nullable columns differed: SQLAlchemy returned Python None in object columns while Arrow used typed float or datetime columns with NaN or NaT, so that shape stayed on the SQLAlchemy path. Developers evaluating an Arrow migration can follow real compatibility gotchas like this on daily.dev."}}]} ```