A step-by-step guide to deploying Mixtral 8x7B on Amazon SageMaker using two optimization techniques: Medusa (speculative decoding) and AWQ (Activation-aware Weight Quantization). Combining both allows deployment on a single g5.12xlarge instance with 4x NVIDIA A10G GPUs, achieving ~50-60ms per token latency and reducing infrastructure costs by ~3x compared to a g5.48xlarge. The guide covers setting up the environment, retrieving the Hugging Face LLM DLC, preparing combined Medusa+AWQ artifacts, deploying the endpoint, and running inference.
