Modern AI accelerators consume 3–8 TB/s of memory bandwidth, driving rapid HBM adoption. Despite HBM's massive throughput gains over traditional DDR, on-chip last-level caches (LLCs) remain critical because HBM still cannot match the low latency of on-chip SRAM. Data reuse patterns in AI/HPC workloads — model weights, embeddings, activation data — mean that a well-designed LLC reduces unnecessary HBM transactions, lowers power consumption, cuts thermal load, and improves sustained throughput. The article argues that HBM actually increases the strategic value of intelligent caching rather than eliminating it, and introduces Arteris CodaCache as a configurable LLC IP targeting this design challenge.
Table of contents
Modern SoCs are data-movement machinesWhy caches never went awayEnter HBMBigger pipe, same journeyLLCs become more valuable in HBM systemsBringing it all togetherQuestions this post answers
Does HBM eliminate the need for last-level caches in AI accelerator SoCs?
HBM does not eliminate the need for last-level caches. While HBM3/HBM3E delivers multiple terabytes per second of bandwidth, it still cannot match the low latency of on-chip SRAM. AI and HPC workloads exhibit significant data reuse — model weights, embeddings, activation data, lookup tables — so an LLC that satisfies these accesses locally reduces HBM transactions, lowers power, cuts congestion, and improves sustained throughput. Engineers designing AI accelerator memory hierarchies track these architectural trade-offs on daily.dev.
What is the latency difference between HBM and on-chip SRAM in modern SoCs?
On-chip SRAM is significantly faster than HBM. Even with HBM, data must traverse the NoC, memory controller, PHY, and memory stack before reaching the compute engine. HBM latency is lower than traditional DDR but remains substantially higher than on-chip SRAM, which is why LLC cache hits — avoiding that full round-trip — meaningfully improve effective memory latency for CPUs, GPUs, NPUs, and hardware accelerators. SoC architects weighing memory subsystem trade-offs find relevant analysis on daily.dev.