--- title: Creepy crawlies | daily.dev description: The maintainer of git.kernel.org shares hard numbers on how AI crawlers are overwhelming the Linux kernel's git hosting infrastructure. Roughly 14-16 CPU cores... canonical: https://daily.dev/posts/creepy-crawlies-w1xrijb39 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Creepy crawlies | daily.dev og:description: The maintainer of git.kernel.org shares hard numbers on how AI crawlers are overwhelming the Linux kernel's git hosting infrastructure. Roughly 14-16 CPU cores... og:url: https://daily.dev/posts/creepy-crawlies-w1xrijb39 og:image: https://api.daily.dev/og/posts/W1XrIJb39.png og:image:alt: Creepy crawlies og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Creepy crawlies **[Lobsters](https://daily.dev/sources/lobsters)** · 8 min read · 0 upvotes · 0 comments ## Summary The maintainer of git.kernel.org shares hard numbers on how AI crawlers are overwhelming the Linux kernel's git hosting infrastructure. Roughly 14-16 CPU cores out of 90, across 5 geo-distributed nodes, are constantly busy rendering git commits as HTML purely for scraper bots training LLMs, consuming about 20% of total capacity. Legitimate traffic is estimated at only 2% of total requests. The bots evolved from identifiable user-agents to residential/mobile IP proxies routed through things like smart TV SDKs, defeating IP bans. Anubis, a proof-of-work challenge system, was deployed and worked for months, but bots eventually started solving even difficulty-5 challenges. The team plans to disable features and gate expensive actions to reduce crawlable URLs, while still promising to offer data downloads to legitimate requesters. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much server capacity is git.kernel.org losing to AI scraper bots? About 20% of total capacity, roughly 14-16 CPU cores out of 90 across 5 geo-distributed nodes, is spent constantly rendering git commits as HTML for scrapers. Legitimate traffic, including git clones, is estimated at only about 2% of total requests, with the rest coming from bots scraping commit data to train language models. _Infrastructure engineers dealing with bot traffic spikes follow real-world mitigation writeups like this on daily.dev._ ### Why do AI crawlers keep bypassing the Anubis proof-of-work challenge on git.kernel.org? Bots kept adapting by simply burning more compute to solve the increasing difficulty levels. Anubis, a proof-of-work system requiring a sha256 hash with leading zeroes, initially blocked scrapers effectively at difficulty 4, but after a few months bots solved it, prompting a raise to difficulty 5, which was also eventually solved by the scrapers while making the challenge noticeably slower for legitimate mobile users. _Teams weighing proof-of-work defenses against determined scrapers track these tradeoffs on daily.dev._ ### Why are AI crawlers using residential and mobile IPs to scrape websites? Crawlers route through residential and mobile IP proxy networks, including SDKs bundled into smart TV apps, to evade IP and ASN-based blocking. Each IP makes only 4-5 requests before disappearing, so banning them is pointless since they never return, making this proxy SDK monetization approach effectively unblockable through conventional firewall rules. _Developers hardening services against distributed scraping compare mitigation strategies on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 3 discussions and 29 comments across lobsters, hackernews (as of 2026-08-31). **TL;DR:** The community largely accepts the numbers as plausible but debates what's actually driving the traffic — training data brokers, vibecoded scrapers, or malicious residential-proxy SDKs — and argues heatedly about which mitigations (proof-of-work, static rendering, firewalling, protocol changes) can actually scale against IP-laundered bot traffic. **Sentiment:** 5% positive · 45% mixed · 50% skeptical **The case for** - Some argue there's a case-by-case, indirection-heavy data broker economy rather than the big AI labs themselves being directly responsible. - One commenter notes LLM-driven traffic might partly reflect legitimate tool-use patterns similar to how humans browse, with real benefits alongside costs. - Several practical mitigation ideas were proposed, like scaling challenge difficulty by content type or fingerprinting via ja3/ja4. **The pushback** - Residential proxies only need bandwidth, not compute, so proof-of-work challenges like Anubis don't meaningfully throttle the actual crawling infrastructure behind them. - Even serving purely static/cheap responses doesn't help when the bottleneck is the TLS handshake itself under high request volume. - Git hosts are inherently crawler traps because they generate huge numbers of combinatorial diff/file links that are extremely expensive to render. - Some see hypocrisy in kernel maintainers fighting AI scrapers while also embracing LLMs for code review and being organizationally tied to companies doing the scraping. - ISPs and app stores have no practical way to identify or block compromised/proxying devices without collateral damage to entire households or networks. **By community** - lobsters (heated): Extensive back-and-forth debating the true source of the traffic (training vs. tool use vs. data brokers) and sharply disagreeing over whether any mitigation—Anubis, fingerprinting, static rendering, firewalling—can actually scale. - hackernews (skeptical): Minimal engagement, with one reply expressing outrage that smart TVs might be proxying scraper traffic. **Hottest debate:** Whether this crawler traffic is truly AI training-related or largely driven by opportunistic data-broker/vibecoded bots, and whether any technical countermeasure can outpace residential-proxy-based crawlers. **Open questions** - What fraction of the traffic comes from malicious embedded SDKs versus botnets of compromised devices? - Is the scraped data actually used for LLM training, or partly for real-time LLM tool-use/browsing? - Can ISPs or app stores realistically be forced to block residential-proxy behavior without collateral damage? **Highlights** > > The only actual scalable solution is to serve static content only and render everything client-side. That helps nothing when your bottleneck is TLS handshake. When you find yourself at the wrong end of a 10k+ request/second scraper wave that lasts days, no matter how efficiently you serve static files, the TLS handshake will kill most cheap VPSes. By cheap, I mean I scaled my normally €4/month VPS at Hetzner up to like €40+/month during a particularly nasty wave, and *that* fell over too. Not even static files, 2 bytes ("OK") + HTTP response headers served from RAM. My solution ended up being firewalling them off. Which I could, even residential proxies, because they're dumb. > — [algernon on lobsters · 1 points](https://lobste.rs/s/nbjo0i/creepy_crawlies#c_8te9qo) > They don't need to have much power at all - only bandwidth, that's why they're *proxies*! The compute part doesn't need to run on the proxy - the proxy only needs to, well, proxy. A bog standard headless chrome in a datacenter can - and does - happily go through a residental proxy. > — [algernon on lobsters · 1 points](https://lobste.rs/s/nbjo0i/creepy_crawlies#c_i4hnks) > I'm dealing with a similar thing at work and it's really annoying. The kernel site is in a worse situation though where they can't run good behaviour analytics, because it's likely someone genuinely comes to the site straight from some external deep link to a decade old commit. Some ideas that could work for them: (or anyone else dealing with this) 1. Scale the difficulty: Static website gets lower than a commit, which gets lower than a custom diff. Frequently requested commit gets lower than an unknown one. If it's not too hard to get the age of a commit pre-check, maybe even vary on that too. 2. Get some fingerprint analysis going. Some crawler networks work around that, but a huge number does not. I won't spell out the details, but... just put things on a dashboard together with ja3/ja4 and you'll see more patterns. Then you can bump the difficulty again (or block) in more targeted ways. 3. Identify some definitely-good traffic that gets lower difficulty. At the extreme, no crawler pretends to be Gnome Web running on Arch ;) 4. Start dropping links to other pages for questionable cases. "If you're a real user, visit the front page to reenable navigation links". > — [viraptor on lobsters · 1 points, 1 comments](https://lobste.rs/s/nbjo0i/creepy_crawlies#c_ufipxu) > There is a new industry of data companies who do this, not the ML model training companies themselves. And then some ML model companies buy the datasets from those, with _that_ level of indirection. As a result, a lot more market players crawling and building their datasets in the hope they'll find a buyer. > — [never\_released on lobsters · 2 points, 1 comments](https://lobste.rs/s/nbjo0i/creepy_crawlies#c_uhwzbt) > They place Anubis in front of kernel.org infra to stop LLMs and then subject people to LLMs for code review and people like Linus and Ted Ts'o engage in moral and environmental whataboutism and "the ends justifies the means" kind of arguments in favor of LLMs. > — [ayushnix on lobsters · 4 points](https://lobste.rs/s/nbjo0i/creepy_crawlies#c_9fqvib) **Source threads** - [lobsters](https://lobste.rs/s/nbjo0i/creepy_crawlies) · 78 points · 25 comments - [hackernews](https://news.ycombinator.com/item?id=49488311) · 9 points · 2 comments - [hackernews](https://news.ycombinator.com/item?id=49491791) · 3 points · 2 comments --- Tags: [#infrastructure](https://daily.dev/tags/infrastructure), [#git](https://daily.dev/tags/git), [#crawling](https://daily.dev/tags/crawling) [View this post on daily.dev](https://daily.dev/posts/creepy-crawlies-w1xrijb39) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Creepy crawlies","url":"https://daily.dev/posts/creepy-crawlies-w1xrijb39","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/creepy-crawlies-w1xrijb39"},"datePublished":"2026-08-29T19:38:02.875Z","dateModified":"2026-08-31T10:51:10.491Z","description":"The maintainer of git.kernel.org shares hard numbers on how AI crawlers are overwhelming the Linux kernel's git hosting infrastructure. Roughly 14-16 CPU cores...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b629e01f02b015897c4b3b0ede2bac93?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b629e01f02b015897c4b3b0ede2bac93?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Lobsters","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Lobsters","logo":"https://media.daily.dev/image/upload/s--tl8v_Fku--/f_auto,t_logo/v1698841318/logos/lobste.jpg","url":"https://daily.dev/sources/lobsters"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/creepy-crawlies-w1xrijb39","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"infrastructure,git,crawling","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Lobsters","item":"https://daily.dev/sources/lobsters"},{"@type":"ListItem","position":3,"name":"Creepy crawlies"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/creepy-crawlies-w1xrijb39#faq","mainEntity":[{"@type":"Question","name":"How much server capacity is git.kernel.org losing to AI scraper bots?","acceptedAnswer":{"@type":"Answer","text":"About 20% of total capacity, roughly 14-16 CPU cores out of 90 across 5 geo-distributed nodes, is spent constantly rendering git commits as HTML for scrapers. Legitimate traffic, including git clones, is estimated at only about 2% of total requests, with the rest coming from bots scraping commit data to train language models. Infrastructure engineers dealing with bot traffic spikes follow real-world mitigation writeups like this on daily.dev."}},{"@type":"Question","name":"Why do AI crawlers keep bypassing the Anubis proof-of-work challenge on git.kernel.org?","acceptedAnswer":{"@type":"Answer","text":"Bots kept adapting by simply burning more compute to solve the increasing difficulty levels. Anubis, a proof-of-work system requiring a sha256 hash with leading zeroes, initially blocked scrapers effectively at difficulty 4, but after a few months bots solved it, prompting a raise to difficulty 5, which was also eventually solved by the scrapers while making the challenge noticeably slower for legitimate mobile users. Teams weighing proof-of-work defenses against determined scrapers track these tradeoffs on daily.dev."}},{"@type":"Question","name":"Why are AI crawlers using residential and mobile IPs to scrape websites?","acceptedAnswer":{"@type":"Answer","text":"Crawlers route through residential and mobile IP proxy networks, including SDKs bundled into smart TV apps, to evade IP and ASN-based blocking. Each IP makes only 4-5 requests before disappearing, so banning them is pointless since they never return, making this proxy SDK monetization approach effectively unblockable through conventional firewall rules. Developers hardening services against distributed scraping compare mitigation strategies on daily.dev."}}]} ```