Step-by-step guide to deploying the Mixtral 8x7B Sparse Mixture of Experts model on AWS Inferentia2 using Hugging Face Optimum and Amazon SageMaker. Covers environment setup, retrieving the TGI Neuronx Deep Learning Container, configuring and deploying the endpoint on an inf2.48xlarge instance, running inference with the Messages API, building a streaming Gradio demo, and benchmarking with llmperf. Benchmark results show 288 tokens/sec throughput at 5 concurrent users with 10.67ms inter-token latency, with cost context of $12.98/hour on-demand for the inf2.48xlarge instance.
