10X more data, same 4 seconds: single-query scaling in Redpanda SQL on 1TB
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Redpanda SQL, a Postgres-compatible analytical engine combining live streaming topics with Apache Iceberg tables, was benchmarked on NYC Taxi trip data scaled from 100 GB to 1 TB (6.07 billion rows). Key findings: time-range queries using Iceberg column statistics for file pruning stay at ~4–5 seconds regardless of total data size (10x growth), while full scans grow linearly from 17s to 181s. Full-scan aggregations scale near-linearly with nodes — a heavy high-cardinality rollup drops from 298s on 1 node to 43s on 8 nodes. Memory for hash-table state also shards across nodes, reducing per-node peak. The practical sizing takeaway: size your cluster for your heaviest full-scan query, not total data volume, since time-bounded dashboard queries are efficient by default via Iceberg pruning.
