Scale LLM Inference on Amazon SageMaker with Multi-Replica Endpoints

A step-by-step guide on deploying multiple replicas of Llama 2 13B on a single Amazon SageMaker endpoint using the new ResourceRequirements object announced at re:Invent 2023. By deploying 8 replicas of Llama 13B on a p4d.24xlarge instance (8x A100 GPUs), throughput is significantly increased. The post covers environment setup, retrieving the Hugging Face LLM DLC, configuring hardware requirements per replica, deploying the model, and running load benchmarks. Benchmark results show the endpoint can handle 500 concurrent users at ~26 req/sec with 100% success rate, though latency grows under heavy load.