--- title: Speeding up ViTs using Block Sparsity | daily.dev description: The post discusses the application of block sparsity on MLP modules in vision transformers, showing promising speedups with minimal accuracy drop. It explains... canonical: https://daily.dev/posts/speeding-up-vits-using-block-sparsity-umq7gw9ns twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Speeding up ViTs using Block Sparsity | daily.dev og:description: The post discusses the application of block sparsity on MLP modules in vision transformers, showing promising speedups with minimal accuracy drop. It explains... og:url: https://daily.dev/posts/speeding-up-vits-using-block-sparsity-umq7gw9ns og:image: https://api.daily.dev/og/posts/umq7gW9NS.png og:image:alt: Speeding up ViTs using Block Sparsity og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Speeding up ViTs using Block Sparsity **[PyTorch](https://daily.dev/sources/pytorch)** · 6 min read · 1 upvotes · 0 comments ## Summary The post discusses the application of block sparsity on MLP modules in vision transformers, showing promising speedups with minimal accuracy drop. It explains the training and inference steps, provides microbenchmarking results, and showcases the speedup and accuracy achieved on a specific ViT model. Future steps and potential optimizations for block sparsity on vision transformers are also mentioned. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: --- Tags: [#deep-learning](https://daily.dev/tags/deep-learning), [#pytorch](https://daily.dev/tags/pytorch) [View this post on daily.dev](https://daily.dev/posts/speeding-up-vits-using-block-sparsity-umq7gw9ns) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Speeding up ViTs using Block Sparsity","url":"https://daily.dev/posts/speeding-up-vits-using-block-sparsity-umq7gw9ns","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/speeding-up-vits-using-block-sparsity-umq7gw9ns"},"datePublished":"2024-05-14T17:16:31.462Z","dateModified":"2024-05-14T17:16:30.341Z","description":"The post discusses the application of block sparsity on MLP modules in vision transformers, showing promising speedups with minimal accuracy drop. It explains...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1e9fe983d3406b3c9ac1a13d24004a42?_a=AQAEuiZ","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1e9fe983d3406b3c9ac1a13d24004a42?_a=AQAEuiZ","isAccessibleForFree":true,"articleSection":"PyTorch","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"PyTorch","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/5fdab3d75a834a97a52d433d9c7e5ff9","url":"https://daily.dev/sources/pytorch"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/speeding-up-vits-using-block-sparsity-umq7gw9ns","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"deep-learning,pytorch","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"PyTorch","item":"https://daily.dev/sources/pytorch"},{"@type":"ListItem","position":3,"name":"Speeding up ViTs using Block Sparsity"}]} ```