Step-by-step guide to deploying Meta's Llama 3 70B Instruct model on Amazon SageMaker using the Hugging Face LLM Deep Learning Container (DLC) powered by Text Generation Inference (TGI). Covers environment setup, hardware requirements for 8B and 70B model sizes (including instance types like ml.p4d.24xlarge), deployment configuration, running inference via the OpenAI-compatible Messages API, and benchmarking with llmperf using 25 concurrent users measuring latency, throughput, and time-to-first-token.
Table of contents
1. Setup development environment2. Hardware requirements3. Deploy Llama 3 to Amazon SageMaker4. Run inference and chat with the model5. Benchmark llama 3 70B with llmperf6. Clean up5 Impressions