--- title: "How to scale LLM workloads to 20B+ with Amazon SageMaker using Hugging Face and PyTorch FSDP" url: https://daily.dev/posts/how-to-scale-llm-workloads-to-20b-with-amazon-sagemaker-using-hugging-face-and-pytorch-fsdp-fvze1bpsb source_url: https://www.philschmid.de/sagemaker-fsdp-gpt type: article source: "Philipp Schmid" published: 2026-05-31T07:41:01.832Z updated: 2026-05-31T08:27:58.012Z tags: ["aws", "llm"] reading_time: 7 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to scale LLM workloads to 20B+ with Amazon SageMaker using Hugging Face and PyTorch FSDP **[Philipp Schmid](https://daily.dev/sources/philschmid)** · 7 min read · 0 upvotes · 0 comments ## Summary Step-by-step guide to fine-tuning a 20B parameter open-source LLM (GPT-NeoXT-Chat-Base-20B) on Amazon SageMaker using PyTorch FSDP and Hugging Face Transformers. Covers environment setup, dataset preparation using the ELI5 dataset with chat formatting, and launching a multi-node (2x p4d.24xlarge with 8x A100 GPUs each) distributed training job via torchrun. Total training cost came to approximately $197 over 2.6 hours. PyTorch FSDP shards model parameters, optimizer states, and gradients across GPUs, enabling training of models too large for a single node. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Scale LLM fine-tuning with Hugging Face and Amazon SageMaker AI](https://daily.dev/posts/scale-llm-fine-tuning-with-hugging-face-and-amazon-sagemaker-ai-4tkqty16y) · AWS · 1 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#llm](https://daily.dev/tags/llm) [View this post on daily.dev](https://daily.dev/posts/how-to-scale-llm-workloads-to-20b-with-amazon-sagemaker-using-hugging-face-and-pytorch-fsdp-fvze1bpsb)