--- title: AI Crawlers Are Bleeding The Linux Kernel Repo's Compute... description: AI scraper bots are consuming roughly 20% of git.kernel.org's total CPU capacity (14-16 of 90 cores across 5 nodes), just to re-render commit pages that are... canonical: https://daily.dev/posts/ai-crawlers-are-bleeding-the-linux-kernel-repo-s-compute-power-95dryensq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AI Crawlers Are Bleeding The Linux Kernel Repo's Compute Power | daily.dev og:description: AI scraper bots are consuming roughly 20% of git.kernel.org's total CPU capacity (14-16 of 90 cores across 5 nodes), just to re-render commit pages that are... og:url: https://daily.dev/posts/ai-crawlers-are-bleeding-the-linux-kernel-repo-s-compute-power-95dryensq og:image: https://api.daily.dev/og/posts/95dryensq.png og:image:alt: AI Crawlers Are Bleeding The Linux Kernel Repo's Compute Power og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI Crawlers Are Bleeding The Linux Kernel Repo's Compute Power **[It's Foss](https://daily.dev/sources/itsfoss)** · 4 min read · 0 upvotes · 0 comments ## Summary AI scraper bots are consuming roughly 20% of git.kernel.org's total CPU capacity (14-16 of 90 cores across 5 nodes), just to re-render commit pages that are already freely cloneable. Linux Foundation sysadmin Konstantin Ryabitsev found that a full clone of linux.git takes 200 CPU-seconds, while scraping the same commits through cgit's individual HTML pages costs 280 CPU-hours, and doing so across all 922 forks balloons to 258,160 CPU-hours. Mitigation attempts including Fail2Ban, ASN blocks, and the Anubis proof-of-work wall have each been defeated as bots adapted. The piece ties this into a broader wave of open source projects (Debian, Rust) setting policies around AI's role in their workflows. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### how much CPU capacity is git.kernel.org losing to AI scrapers About 20% of git.kernel.org's total compute capacity, or 14 to 16 of its 90 CPU cores spread across 5 geo-distributed nodes, is constantly spent rendering commit pages for scrapers rather than serving real developers, according to Linux Foundation sysadmin Konstantin Ryabitsev. The actual load is spikier than a flat 20%, arriving in waves as bot swarms descend on the server. _Anyone tracking how AI crawlers strain open source infrastructure can follow ongoing coverage on daily.dev._ ### why is scraping git.kernel.org through cgit so much more expensive than cloning the repo Cloning linux.git locally, which pulls the entire commit history, takes about 200 CPU-seconds of server time, but scraping the same 1.48 million commits through cgit's individual HTML pages costs 280 CPU-hours instead. Scaled across all 922 forks hosted on the server, that becomes 258,160 CPU-hours, roughly 4.6 million times more expensive than a single clone. _Developers weighing repo hosting and bot-mitigation tradeoffs can track this kind of infrastructure story on daily.dev._ ### what defenses have been tried against AI scrapers hitting kernel.org Fail2Ban and IP bans worked initially but stopped once bots spread across entire subnets, and ASN blocks worked briefly until millions of residential and mobile IPs took over scraping instead. The Anubis proof-of-work challenge wall came next and worked for a while, until the bots started solving increasing levels of its difficulty. _Teams building anti-scraping defenses can keep up with how these arms races evolve via daily.dev._ --- Tags: [#open-source](https://daily.dev/tags/open-source), [#linux](https://daily.dev/tags/linux), [#crawling](https://daily.dev/tags/crawling) [View this post on daily.dev](https://daily.dev/posts/ai-crawlers-are-bleeding-the-linux-kernel-repo-s-compute-power-95dryensq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AI Crawlers Are Bleeding The Linux Kernel Repo's Compute Power","url":"https://daily.dev/posts/ai-crawlers-are-bleeding-the-linux-kernel-repo-s-compute-power-95dryensq","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/ai-crawlers-are-bleeding-the-linux-kernel-repo-s-compute-power-95dryensq"},"datePublished":"2026-08-31T10:49:55.951Z","dateModified":"2026-08-31T10:51:10.488Z","description":"AI scraper bots are consuming roughly 20% of git.kernel.org's total CPU capacity (14-16 of 90 cores across 5 nodes), just to re-render commit pages that are...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2c949f64c560848f5678e2d656f8310b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2c949f64c560848f5678e2d656f8310b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"It's Foss","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"It's Foss","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/582e3faed71842bfb9d3840a787d6700","url":"https://daily.dev/sources/itsfoss"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/ai-crawlers-are-bleeding-the-linux-kernel-repo-s-compute-power-95dryensq","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"open-source,linux,crawling","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"It's Foss","item":"https://daily.dev/sources/itsfoss"},{"@type":"ListItem","position":3,"name":"AI Crawlers Are Bleeding The Linux Kernel Repo's Compute Power"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/ai-crawlers-are-bleeding-the-linux-kernel-repo-s-compute-power-95dryensq#faq","mainEntity":[{"@type":"Question","name":"how much CPU capacity is git.kernel.org losing to AI scrapers","acceptedAnswer":{"@type":"Answer","text":"About 20% of git.kernel.org's total compute capacity, or 14 to 16 of its 90 CPU cores spread across 5 geo-distributed nodes, is constantly spent rendering commit pages for scrapers rather than serving real developers, according to Linux Foundation sysadmin Konstantin Ryabitsev. The actual load is spikier than a flat 20%, arriving in waves as bot swarms descend on the server. Anyone tracking how AI crawlers strain open source infrastructure can follow ongoing coverage on daily.dev."}},{"@type":"Question","name":"why is scraping git.kernel.org through cgit so much more expensive than cloning the repo","acceptedAnswer":{"@type":"Answer","text":"Cloning linux.git locally, which pulls the entire commit history, takes about 200 CPU-seconds of server time, but scraping the same 1.48 million commits through cgit's individual HTML pages costs 280 CPU-hours instead. Scaled across all 922 forks hosted on the server, that becomes 258,160 CPU-hours, roughly 4.6 million times more expensive than a single clone. Developers weighing repo hosting and bot-mitigation tradeoffs can track this kind of infrastructure story on daily.dev."}},{"@type":"Question","name":"what defenses have been tried against AI scrapers hitting kernel.org","acceptedAnswer":{"@type":"Answer","text":"Fail2Ban and IP bans worked initially but stopped once bots spread across entire subnets, and ASN blocks worked briefly until millions of residential and mobile IPs took over scraping instead. The Anubis proof-of-work challenge wall came next and worked for a while, until the bots started solving increasing levels of its difficulty. Teams building anti-scraping defenses can keep up with how these arms races evolve via daily.dev."}}]} ```