A practical guide to fine-tuning Vision-Language Models (VLMs) using Hugging Face TRL, Transformers, and Datasets. Covers the full workflow: defining a multimodal use case (product description generation from images), setting up the environment, preparing a conversational dataset, fine-tuning with SFTTrainer and QLoRA on a 24GB GPU, and evaluating results. Uses Qwen2-VL-7B as the primary example but the approach works with Llama 3.2 Vision, Pixtral, and others. Training ~1k samples took under 2 hours at ~$1.40 on an AWS g6.2xlarge instance. Also covers merging LoRA adapters back into the base model for deployment.
