Step-by-step guide to deploying Falcon 180B, a 180-billion-parameter open source LLM, on Amazon SageMaker using the Hugging Face LLM Deep Learning Container (DLC) powered by Text Generation Inference (TGI). Covers environment setup, retrieving the DLC image URI, hardware requirements (p4de.24xlarge with 8 A100 GPUs), deploying the model endpoint, running chat inference with configurable parameters, and building a streaming Gradio chatbot demo using SageMaker's Server-Sent-Events streaming support.
Table of contents
1. Setup development environment2. Retrieve the new Hugging Face LLM DLC3. Hardware requirements4. Deploy Falcon 180B to Amazon SageMaker5. Run inference and chat with the model6. Create a streaming Gradio Chatbot Demo3 Impressions