--- title: "Fine-tune Falcon 180B with DeepSpeed ZeRO, LoRA & Flash Attention" url: https://daily.dev/posts/fine-tune-falcon-180b-with-deepspeed-zero-lora-flash-attention-rxhebhaqc source_url: https://www.philschmid.de/deepspeed-lora-flash-attention type: article source: "Philipp Schmid" published: 2026-05-31T07:41:31.394Z updated: 2026-05-31T08:08:43.173Z tags: ["llm", "lora"] reading_time: 6 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Fine-tune Falcon 180B with DeepSpeed ZeRO, LoRA & Flash Attention **[Philipp Schmid](https://daily.dev/sources/philschmid)** · 6 min read · 0 upvotes · 0 comments ## Summary Step-by-step guide to fine-tuning Falcon 180B, the largest open-source LLM with 180B parameters, on a multi-GPU machine using DeepSpeed ZeRO stage 3, LoRA via Hugging Face PEFT, and Flash Attention 2. Covers environment setup, dataset preparation with the Dolly dataset, instruction formatting, sample packing, and launching distributed training with torchrun. The combination of techniques reduces memory usage dramatically, enabling ~2-hour fine-tuning runs on 8x A100 80GB GPUs compared to the 7 million GPU hours required for pretraining. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Fine-Tuning NVIDIA Cosmos Predict 2.5 with LoRA/DoRA for Robot Video Generation](https://daily.dev/posts/fine-tuning-nvidia-cosmos-predict-2-5-with-lora-dora-for-robot-video-generation-uptnktldn) · Hugging Face · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#lora](https://daily.dev/tags/lora) [View this post on daily.dev](https://daily.dev/posts/fine-tune-falcon-180b-with-deepspeed-zero-lora-flash-attention-rxhebhaqc)