A performance benchmark study of a 56-node CrateDB cluster on Microsoft Azure, achieving up to 4.8 million records per second ingestion. The post covers the hardware setup (D12v2/D14v2 VMs with SSD storage), bulk insert strategies using Python load generators, shard configuration tuning (optimal ratio of ~2 shards per node), memory and I/O settings (heap allocation, mmapfs, store throttle), and the impact of horizontal scaling on write throughput. Key findings: doubling data nodes from 14 to 28 nearly doubled ingestion rates, and disabling replication and refresh during bulk loads significantly improves performance.
Table of contents
SetupPump it up!Merge larger segmentsIncreasing the number of data nodesIn search of key factorsConclusion348 Impressions