Delta encoding stores successive differences between values rather than absolute values, making time-series data far more compressible. Using a real dataset of 200,102 samples from an embedded system, the post demonstrates how applying delta encoding to timestamps before saving as Apache Feather (Arrow) with zstd compression reduces file size from ~2.96MB (raw CSV) to ~455KB — a 64% reduction in timestamp storage. The key insight is that nearly-regular timestamps produce repeated small delta values that compress extremely well. The post includes Shannon entropy calculations to validate the theoretical compression ratio, shows how to encode/decode with NumPy diff and cumsum, and discusses int32 downcasting and millisecond-resolution quantization as further optimizations. A plea is made for native Feather/Pandas library support for delta encoding metadata.
