A deep-dive into three interconnected layers of storage I/O performance: disk, filesystem, and application. At the disk layer, NVMe firmware misreporting physical sector sizes (512B vs 4K) caused up to 42% fewer measured write IOPS on AWS i7i instances, and RAID0 chunk/stripe alignment can unevenly distribute load. At the filesystem layer, formatting XFS with 1K block sizes on 4K-optimized NVMe forced read-modify-write operations on ~75% of requests; switching to 4K blocks restored full throughput. Metadata operations (extent allocation, inode updates, ctime flushes) can block async I/O submissions even with RWF_NOWAIT; pre-truncating or pre-allocating files with fallocate()/ftruncate() mitigates this. At the application layer, Little's Law governs the queue depth needed to saturate device bandwidth, and small I/O request sizes hit IOPS/CPU ceilings before PCIe/NAND bandwidth limits. Practical guidance includes verifying sector sizes independently of firmware, matching filesystem block size to physical sector size, pre-allocating files, and tuning parallelism and request sizes via benchmarking.
