--- title: GitHub - Saivineeth147/lora-speedrun: Speedrunning LoRA... description: LoRA Speedrun is an open competition for optimizing LoRA fine-tuning speed on a fixed task and hardware setup, inspired by the modded-nanogpt pretraining... canonical: https://daily.dev/posts/github---saivineeth147-lora-speedrun-speedrunning-lora-fine-tuning-frozen-task-frozen-hardware-p-aivec6okz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: GitHub - Saivineeth147/lora-speedrun: Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning. | daily.dev og:description: LoRA Speedrun is an open competition for optimizing LoRA fine-tuning speed on a fixed task and hardware setup, inspired by the modded-nanogpt pretraining... og:url: https://daily.dev/posts/github---saivineeth147-lora-speedrun-speedrunning-lora-fine-tuning-frozen-task-frozen-hardware-p-aivec6okz og:image: https://api.daily.dev/og/posts/aIVEC6okz.png og:image:alt: GitHub - Saivineeth147/lora-speedrun: Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # GitHub - Saivineeth147/lora-speedrun: Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning. **[Hacker News](https://daily.dev/sources/hn)** · 7 min read · 1 upvotes · 0 comments ## Summary LoRA Speedrun is an open competition for optimizing LoRA fine-tuning speed on a fixed task and hardware setup, inspired by the modded-nanogpt pretraining speedrun. Two frozen tracks exist: Track 1 fine-tunes Qwen2.5-1.5B to ≥57% on GSM8K math, and Track 2 fine-tunes SmolLM2-1.7B to ≥75.5% on SQuAD v1.1 QA — both on a single L40S GPU via Modal. The current record is 6m 05s (down from an 11m 57s baseline), achieved via sequence packing and completion-only loss masking. Competing and verifying are free using Modal's monthly credits. Records require 3 independent re-runs with fresh seeds. The project aims to create an apples-to-apples public arena for comparing LoRA techniques like DoRA, rsLoRA, PiSSA, NEFTune, and Unsloth kernels that currently lack comparable benchmarks. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the current fastest verified time to LoRA fine-tune Qwen2.5-1.5B to 57% on GSM8K? The current record is 6 minutes 5 seconds, achieved using sequence packing combined with completion-only loss masking over 2 epochs, using the same LoRA configuration as the baseline record but running roughly twice as fast while also reaching higher accuracy. All records are independently re-run three times with fresh seeds on identical single-GPU hardware before they count. _Track evolving LoRA speed techniques like this one on daily.dev as you optimize your own fine-tuning pipeline._ ### Why does this LoRA fine-tuning benchmark use an L40S GPU on Modal instead of an H100 or 4090? An L40S on Modal is used because it is one consistent datacenter SKU making times genuinely comparable, unlike rented consumer cards which vary host to host. It is also free to use via Modal's monthly compute credits, and Modal sandboxes run untrusted submitted code network-blocked and secretless. The L40S shares the same AD102 silicon as the 4090, so consumer-GPU tricks still transfer. _Compare GPU benchmarking tradeoffs like this on daily.dev when picking hardware for fine-tuning workloads._ ### What LoRA fine-tuning techniques are allowed or banned in a wall-clock speedrun benchmark like this? Allowed techniques include LoRA rank and placement choices, quantization, learning-rate schedules, sequence packing, data subset selection, custom kernels, and early stopping, all within an adapter-only constraint of 30 million or fewer trainable parameters. Training on data outside the official train split, distilling from teacher models, or using synthetic data is explicitly banned to keep results comparable. _Keep up with which fine-tuning tricks actually hold up under fair comparison via daily.dev._ ## Similar posts on daily.dev - [Unsloth and Training Hub: Lightning-fast LoRA and QLoRA fine-tuning](https://daily.dev/posts/unsloth-and-training-hub-lightning-fast-lora-and-qlora-fine-tuning-nhdohkubf) · Red Hat Developer · 1 upvotes · 0 comments - [Fine-Tune an Open-Source LLM on Your Own Data](https://daily.dev/posts/fine-tune-an-open-source-llm-on-your-own-data-fl1zitmwk) · Medium · 2 upvotes · 0 comments - [How to Fine-Tune an LLM: An End-to-End Guide](https://daily.dev/posts/how-to-fine-tune-an-llm-an-end-to-end-guide-dgv3ruc1u) · Towards Data Science · 0 upvotes · 0 comments - [Fine-Tuning NVIDIA Cosmos Predict 2.5 with LoRA/DoRA for Robot Video Generation](https://daily.dev/posts/fine-tuning-nvidia-cosmos-predict-2-5-with-lora-dora-for-robot-video-generation-uptnktldn) · Hugging Face · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#llm](https://daily.dev/tags/llm), [#deep-learning](https://daily.dev/tags/deep-learning) [View this post on daily.dev](https://daily.dev/posts/github---saivineeth147-lora-speedrun-speedrunning-lora-fine-tuning-frozen-task-frozen-hardware-p-aivec6okz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"GitHub - Saivineeth147/lora-speedrun: Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.","url":"https://daily.dev/posts/github---saivineeth147-lora-speedrun-speedrunning-lora-fine-tuning-frozen-task-frozen-hardware-p-aivec6okz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/github---saivineeth147-lora-speedrun-speedrunning-lora-fine-tuning-frozen-task-frozen-hardware-p-aivec6okz"},"datePublished":"2026-07-20T09:37:40.201Z","dateModified":"2026-09-13T21:42:14.345Z","description":"LoRA Speedrun is an open competition for optimizing LoRA fine-tuning speed on a fixed task and hardware setup, inspired by the modded-nanogpt pretraining...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/cbc08510d9801d366b92d9c27e3a627e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/cbc08510d9801d366b92d9c27e3a627e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/github---saivineeth147-lora-speedrun-speedrunning-lora-fine-tuning-frozen-task-frozen-hardware-p-aivec6okz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,llm,deep-learning","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"GitHub - Saivineeth147/lora-speedrun: Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/github---saivineeth147-lora-speedrun-speedrunning-lora-fine-tuning-frozen-task-frozen-hardware-p-aivec6okz#faq","mainEntity":[{"@type":"Question","name":"What is the current fastest verified time to LoRA fine-tune Qwen2.5-1.5B to 57% on GSM8K?","acceptedAnswer":{"@type":"Answer","text":"The current record is 6 minutes 5 seconds, achieved using sequence packing combined with completion-only loss masking over 2 epochs, using the same LoRA configuration as the baseline record but running roughly twice as fast while also reaching higher accuracy. All records are independently re-run three times with fresh seeds on identical single-GPU hardware before they count. Track evolving LoRA speed techniques like this one on daily.dev as you optimize your own fine-tuning pipeline."}},{"@type":"Question","name":"Why does this LoRA fine-tuning benchmark use an L40S GPU on Modal instead of an H100 or 4090?","acceptedAnswer":{"@type":"Answer","text":"An L40S on Modal is used because it is one consistent datacenter SKU making times genuinely comparable, unlike rented consumer cards which vary host to host. It is also free to use via Modal's monthly compute credits, and Modal sandboxes run untrusted submitted code network-blocked and secretless. The L40S shares the same AD102 silicon as the 4090, so consumer-GPU tricks still transfer. Compare GPU benchmarking tradeoffs like this on daily.dev when picking hardware for fine-tuning workloads."}},{"@type":"Question","name":"What LoRA fine-tuning techniques are allowed or banned in a wall-clock speedrun benchmark like this?","acceptedAnswer":{"@type":"Answer","text":"Allowed techniques include LoRA rank and placement choices, quantization, learning-rate schedules, sequence packing, data subset selection, custom kernels, and early stopping, all within an adapter-only constraint of 30 million or fewer trainable parameters. Training on data outside the official train split, distilling from teacher models, or using synthetic data is explicitly banned to keep results comparable. Keep up with which fine-tuning tricks actually hold up under fair comparison via daily.dev."}}]} ```