Materialize uses dictionary compression to reduce peak memory usage by more than half without slowing data hydration. The approach leverages unused tag bytes in Materialize's row representation to encode the most frequent column values as single bytes. Frequent values are identified on-the-fly using the Misra-Gries streaming algorithm, which maintains approximate counts of the most common elements during data loading. Empirical results on a ~1TB internal knowledge graph workload show peak memory halved, enabling a 2x resource reduction, though further halving causes failures when compressed data no longer fits available resources.
Table of contents
Materialize's Row RepresentationBuilding a DictionaryCompression ResultsConclusions134 Impressions