A practical guide to fine-tuning and deploying Mistral 7B using Hugging Face on Amazon SageMaker. Covers dataset preparation with the Dolly dataset, QLoRA-based fine-tuning with 4-bit quantization and LoRA adapters, Flash Attention 2 for faster training, and deployment via the Hugging Face LLM Inference DLC with Text Generation Inference. Also demonstrates response streaming with a Gradio app. Total training cost on an ml.g5.4xlarge instance was approximately $8 for ~3.9 hours.
Table of contents
1. Setup Development Environment2. Load and prepare the dataset3. Fine-Tune Mistral 7B with QLoRA on Amazon SageMaker4. Deploy Fine-tuned Mistral 7B on Amazon SageMaker5. Stream Inference Requests from the Deployed Model2 Impressions