--- title: A primer on GPUs and kernel optimization | daily.dev description: This is the first of a three-part series documenting GPU kernel optimization work from a custom RL framework called Jam, covering GPU architecture, distributed... canonical: https://daily.dev/posts/a-primer-on-gpus-and-kernel-optimization-qxwmpwwhr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: A primer on GPUs and kernel optimization | daily.dev og:description: This is the first of a three-part series documenting GPU kernel optimization work from a custom RL framework called Jam, covering GPU architecture, distributed... og:url: https://daily.dev/posts/a-primer-on-gpus-and-kernel-optimization-qxwmpwwhr og:image: https://api.daily.dev/og/posts/QXWmpWWhr.png og:image:alt: A primer on GPUs and kernel optimization og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # A primer on GPUs and kernel optimization **[Hacker News](https://daily.dev/sources/hn)** · 42 min read · 0 upvotes · 0 comments ## Summary This is the first of a three-part series documenting GPU kernel optimization work from a custom RL framework called Jam, covering GPU architecture, distributed computation, and profiling basics before walking through real kernel optimizations. It explains data types, warps, SMs, occupancy, and roofline modeling, then applies those ideas with concrete traces. Examples include a paged-attention indexing fix that drops a kernel from 797.16 to 507.43 µs by simplifying address math, a zigzag ring-attention rebalancing that cuts median GPU time 19.3%, a parallel MoE route-grouping change giving a 16.4x speedup on sorting, and an FP8-to-FP16 conversion trick cutting a paged-attention kernel from 183 to 138.82 µs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How many SMs does an NVIDIA H200 SXM GPU have? A full H200 SXM has 132 streaming multiprocessors (SMs). Each SM contains arithmetic hardware and storage for the threads it runs, and the GPU schedules thread blocks across all of these SMs during a kernel launch, with later blocks waiting if there aren't enough SM resources available immediately. _Anyone sizing kernel launches or occupancy on H200 hardware can track GPU architecture deep dives like this on daily.dev._ ### What is the HBM memory bandwidth of an H200 GPU and how do I estimate a kernel's minimum runtime from it? An H200 SXM delivers about 4.8 TB/s of HBM bandwidth. To estimate a lower bound on kernel runtime, divide the bytes that must be read or written by this bandwidth; for example, reading 1 GB from HBM takes about 208 microseconds. Compare this against the compute-bound estimate (FLOPs divided by peak throughput, about 990 TFLOP/s for dense BF16) and take the larger of the two as your roofline estimate. _Developers estimating kernel speed-of-light targets before optimizing can find hardware-specific breakdowns like this on daily.dev._ ### Why would replacing an array sort with a custom per-expert kernel speed up Mixture of Experts (MoE) routing on a GPU? Because JAX's lax.sort for grouping MoE routes by expert ran as a single block of 1,024 threads on only one SM, leaving 131 other SMs idle. Giving each expert its own block (129 blocks of 128 threads) spreads the work across SMs using a prefix sum per block, cutting route-grouping time from 36.70 to 2.24 microseconds, a 16.4x speedup, and reducing the full BF16 MoE stage time by about 13%. _Engineers debugging underutilized SMs in MoE routing kernels can compare optimization approaches like this on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 15 comments across hackernews (as of 2026-11-10). **TL;DR:** Discussion centers on questions about how to properly benchmark and validate kernel optimizations, alongside praise for the writing quality, with a side debate over whether the post is about autonomous agent hillclimbing or human-guided optimization. **Sentiment:** 35% positive · 45% mixed · 20% skeptical **The case for** - The writing is seen as unusually clear and well-explained. - Some appreciated the practical framing of numerics tolerance differing between training and inference use cases. **The pushback** - Some felt the post is missing discussion of benchmarking/evaluation methodology for the kernels (cache state, thermal noise, run-to-run variance, launch overhead). - A minor technical nit was raised that HBM is off-chip memory, contradicting a claim in the post. - Some suspect the page design and possibly the writing were AI-generated/templated rather than fully human. **By community** - hackernews (mixed): People debate whether rigorous benchmarking methodology was missing, whether the post concerns human-guided or autonomous optimization, and whether the writing/design was AI-assisted, alongside a technical correction about HBM. **Hottest debate:** Whether the post's lack of benchmarking/evaluation rigor is a genuine gap or irrelevant because the work is human-guided rather than autonomous hillclimbing. **Open questions** - How should kernel benchmarks account for thermal noise, cache state, and run-to-run variance? - What error margins or equivalence thresholds should a kernel be held to versus a reference implementation? - Is task-based end-to-end evaluation sufficient, or is per-op numerical validation also needed? **Highlights** > Crazy to not see a word about benchmarking/evaluation of the kernels. IMO at this time when all performant GPU kernels are generated/hill-climbed, its the largest source of error we see. Should be kernel run with a hot or cold cache in practice? Should your kernel be bitwise equivalent to a ref? What error margins should your kernel have to ref? How many times are you running the kernel to avoid thermal noise? Are you running on separate GPUs to avoid overfitting the current hardware? Can you cudagraph-compiling your kernel to remove launch overhead? Are you A/B B/A interleaving your kernel/reference runs to avoid running thermal bias? > — [augment\_me on hackernews · 2 comments](https://news.ycombinator.com/item?id=50432557) > It’s not crazy at all, the blog post is clearly about human-guided optimization with LLMs rather than hillclimbing with an autonomous agent. > — [saagarjha on hackernews · 1 comments](https://news.ycombinator.com/item?id=50434718) > Yes, but they have nothing to do with GPUs and kernels. Those are general performance standards that would apply to any effort regardless of target. > — [saagarjha on hackernews · 1 comments](https://news.ycombinator.com/item?id=50434861) > > On the H200s used for most of this work, the GPU's main memory is HBM, or high-bandwidth memory. Ironically, this is the slowest memory on the chip. Nit: HBM is off chip, unlike shared memory and registers > — [saagarjha on hackernews](https://news.ycombinator.com/item?id=50434738) > The writing here is so clear. Do you use Llms to write? > — [kollegekid on hackernews · 2 comments](https://news.ycombinator.com/item?id=50433417) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50386246) · 144 points · 15 comments ## Similar posts on daily.dev - [A Guide to Understanding GPUs and Maximizing GPU Utilization](https://daily.dev/posts/a-guide-to-understanding-gpus-and-maximizing-gpu-utilization-b9b9tbntz) · Towards Data Science · 0 upvotes · 0 comments - [How Work is Organized Inside a GPU](https://daily.dev/posts/how-work-is-organized-inside-a-gpu-jugfqh8ty) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments - [CUDA Guide: Workflow for Performance Tuning](https://daily.dev/posts/cuda-guide-workflow-for-performance-tuning-wurjlraei) · DigitalOcean Community · 1 upvotes · 0 comments - [Systems Engineering Playbook: Optimizing Qwen 3.5-397B MoE on Ironwood \(TPU7x\)](https://daily.dev/posts/systems-engineering-playbook-optimizing-qwen-3-5-397b-moe-on-ironwood-tpu7x--quc6gr0fd) · Google Developers · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#cuda](https://daily.dev/tags/cuda), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts) [View this post on daily.dev](https://daily.dev/posts/a-primer-on-gpus-and-kernel-optimization-qxwmpwwhr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"A primer on GPUs and kernel optimization","url":"https://daily.dev/posts/a-primer-on-gpus-and-kernel-optimization-qxwmpwwhr","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/a-primer-on-gpus-and-kernel-optimization-qxwmpwwhr"},"datePublished":"2026-11-09T22:48:28.883Z","dateModified":"2026-11-10T09:34:21.787Z","description":"This is the first of a three-part series documenting GPU kernel optimization work from a custom RL framework called Jam, covering GPU architecture, distributed...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/bb41eb189c9d92479b57523de5e4a613?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/bb41eb189c9d92479b57523de5e4a613?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/a-primer-on-gpus-and-kernel-optimization-qxwmpwwhr","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,cuda,mixture-of-experts","timeRequired":"PT42M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"A primer on GPUs and kernel optimization"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/a-primer-on-gpus-and-kernel-optimization-qxwmpwwhr#faq","mainEntity":[{"@type":"Question","name":"How many SMs does an NVIDIA H200 SXM GPU have?","acceptedAnswer":{"@type":"Answer","text":"A full H200 SXM has 132 streaming multiprocessors (SMs). Each SM contains arithmetic hardware and storage for the threads it runs, and the GPU schedules thread blocks across all of these SMs during a kernel launch, with later blocks waiting if there aren't enough SM resources available immediately. Anyone sizing kernel launches or occupancy on H200 hardware can track GPU architecture deep dives like this on daily.dev."}},{"@type":"Question","name":"What is the HBM memory bandwidth of an H200 GPU and how do I estimate a kernel's minimum runtime from it?","acceptedAnswer":{"@type":"Answer","text":"An H200 SXM delivers about 4.8 TB/s of HBM bandwidth. To estimate a lower bound on kernel runtime, divide the bytes that must be read or written by this bandwidth; for example, reading 1 GB from HBM takes about 208 microseconds. Compare this against the compute-bound estimate (FLOPs divided by peak throughput, about 990 TFLOP/s for dense BF16) and take the larger of the two as your roofline estimate. Developers estimating kernel speed-of-light targets before optimizing can find hardware-specific breakdowns like this on daily.dev."}},{"@type":"Question","name":"Why would replacing an array sort with a custom per-expert kernel speed up Mixture of Experts (MoE) routing on a GPU?","acceptedAnswer":{"@type":"Answer","text":"Because JAX's lax.sort for grouping MoE routes by expert ran as a single block of 1,024 threads on only one SM, leaving 131 other SMs idle. Giving each expert its own block (129 blocks of 128 threads) spreads the work across SMs using a prefix sum per block, cutting route-grouping time from 36.70 to 2.24 microseconds, a 16.4x speedup, and reducing the full BF16 MoE stage time by about 13%. Engineers debugging underutilized SMs in MoE routing kernels can compare optimization approaches like this on daily.dev."}}]} ```