Deploying large LLMs on AWS GPU instances involves painful cold-start waits as model weights load into GPU HBM. By combining Amazon FSx for Lustre with NVIDIA GPUDirect Storage (GDS) and pre-sharded, pre-quantized (FP8) model checkpoints, Llama 3.1 405B load times drop from 10–20 minutes to just 6.4 seconds — a ~169x speedup. The approach bypasses CPU memory entirely: all 8 GPUs read their tensor-parallel shards in parallel directly from FSx for Lustre into HBM via EFA. Measured on a P5en (8x H200) with a 96 TiB Persistent_2 EFA filesystem, the technique eliminates CPU bounce buffers, serial deserialization, and sequential GPU loading. Additionally, applying TurboQuant KV cache compression (3–4 bits per value) expands the usable context window from ~82K tokens to over 400K tokens on a P5en, or ~660K tokens on a P6 node — a 5x increase on the same hardware. The pattern integrates with vLLM and TensorRT-LLM and scales linearly with filesystem capacity.
