Step-by-step guide to fine-tuning a 20B parameter open-source LLM (GPT-NeoXT-Chat-Base-20B) on Amazon SageMaker using PyTorch FSDP and Hugging Face Transformers. Covers environment setup, dataset preparation using the ELI5 dataset with chat formatting, and launching a multi-node (2x p4d.24xlarge with 8x A100 GPUs each) distributed training job via torchrun. Total training cost came to approximately $197 over 2.6 hours. PyTorch FSDP shards model parameters, optimizer states, and gradients across GPUs, enabling training of models too large for a single node.
