--- title: University libraries are being swarmed by AI bots,... description: AI scraping bots are overwhelming university library websites, sometimes knocking entire collections offline. The University of North Carolina at Chapel Hill... canonical: https://daily.dev/posts/university-libraries-are-being-swarmed-by-ai-bots-putting-public-access-at-risk-ivxtowrzf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: University libraries are being swarmed by AI bots, putting public access at risk | daily.dev og:description: AI scraping bots are overwhelming university library websites, sometimes knocking entire collections offline. The University of North Carolina at Chapel Hill... og:url: https://daily.dev/posts/university-libraries-are-being-swarmed-by-ai-bots-putting-public-access-at-risk-ivxtowrzf og:image: https://api.daily.dev/og/posts/iVXtowRZf.png og:image:alt: University libraries are being swarmed by AI bots, putting public access at risk og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # University libraries are being swarmed by AI bots, putting public access at risk **[Hacker News](https://daily.dev/sources/hn)** · 6 min read · 1 upvotes · 0 comments ## Summary AI scraping bots are overwhelming university library websites, sometimes knocking entire collections offline. The University of North Carolina at Chapel Hill got over 11,000 searches for Finnish music from thousands of IP addresses in a single day. These bots sometimes ignore robots.txt rules and in some cases request thousands of documents per second, unlike compliant crawlers from Google or Internet Archive. Since one bot farm can use thousands of IP addresses, blocking individual IPs is no longer useful. Anthropic, OpenAI, Google and Microsoft recently expressed support for honouring a standardized robots.txt opt-out signal, but compliance is voluntary and it excludes third-party scrapers and smaller AI startups. Some libraries use CAPTCHAs, which can also weed out good bots, and some require login credentials, which those interviewed want to avoid. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why can't libraries just block the IP addresses of AI scraping bots Blocking individual IP addresses no longer works because a single company or bot farm can operate from thousands of different internet addresses simultaneously. Eric Hellman of Project Gutenberg notes this makes the old IP-blocking fix ineffective, forcing institutions toward other measures like CAPTCHAs or login requirements, which also end up blocking legitimate human users and compliant bots. _Teams fighting scraper traffic compare mitigation approaches and outcomes on daily.dev._ ### what did OpenAI, Google, Anthropic and Microsoft agree on for AI web scraping opt-outs These companies expressed support for honoring a single, standardized opt-out signal placed in a website's robots.txt file that would block their web crawlers from harvesting data. However, compliance is voluntary, and the agreement does not cover problematic bots run by third-party companies or smaller AI startups, leaving a significant gap in protection. _Developers tracking AI data-policy shifts follow standards like this one on daily.dev._ ### how bad is AI bot traffic for library and archive websites It is severe enough to periodically crash sites entirely. The University of North Carolina at Chapel Hill reported over 11,000 searches for Finnish music from thousands of distinct IP addresses in a single day, and AI scraping bots have been recorded requesting thousands of documents per second while downloading every available file format, unlike compliant crawlers that follow robots.txt. _Infrastructure engineers sizing capacity against scraper surges can find similar case reports on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 2 discussions and 48 comments across hackernews (as of 2026-11-06). **TL;DR:** Commenters mostly debate practical mitigations (rate limiting, fingerprinting, IP lists, proof-of-work) for AI scraping, while a sizable faction argues the real story is libraries' outdated, poorly architected infrastructure rather than the bots themselves. **Sentiment:** 5% positive · 35% mixed · 60% skeptical **The case for** - Some argue bots and scrapers ultimately serve real human users (e.g. liberating public-domain scans, buying products), so blocking them has downsides. - A few note LLM-bot traffic itself is relatively low-cost and will likely be self-throttled by responsible providers like OpenAI/Anthropic. **The pushback** - Several argue the scale (11,000 searches/day) isn't actually that large and reveals the libraries' infrastructure and architecture are simply inadequate. - Blocking by IP is increasingly useless since bot farms rotate across thousands of residential proxy IPs. - CAPTCHAs, logins, and other friction-heavy defenses degrade access for legitimate human users. - Many non-LLM scrapers (data brokers, debt collectors, legal services) are also hammering public sites, not just AI companies. - There's concern that legal accountability for damaging scraping is weak or inconsistently enforced, citing CFAA case law and alleged unauthorized access by AI companies. **By community** - hackernews (heated): Discussion splits between detailed technical mitigation advice and a contentious argument over whether libraries themselves are to blame for poor scalability rather than the bots. **Hottest debate:** Whether the core problem is irresponsible AI bot traffic or that university libraries have badly designed, under-resourced systems unable to handle any real-world load. **Open questions** - Who exactly is running these bot farms — major AI labs, smaller startups, or data-broker-style contractors reselling scraped data? - What technical and legal remedies actually work against IP-rotating bot farms at scale? - How can sites distinguish between legitimate AI-driven human requests and abusive mass scraping without excluding real users? **Highlights** > Agreed, this just reads like the universities have no idea how to actually serve data effectively. Just bad architecture and lazy or incompetent administrators. > — [robcohen on hackernews · 2 comments](https://news.ycombinator.com/item?id=50393355) > In what world is 11k searches a lot? The story here is that AI bots are exposing University Libraries bad setups. I haven't heard any institution here in northern europe having any trouble. > — [tokai on hackernews · 2 comments](https://news.ycombinator.com/item?id=50393093) > I would encourage you to re-think your comments. You just labelled some people "lazy or incompetent" while having barely any knowledge of the situation, or what resources they have or don't have available. Libraries aren't start ups focusing on scalability, their focus is on preserving knowledge and helping academics and the public find it. If you believe that they should focus on the scalability required to handle bot traffic perhaps you would be best donating money to them to help give them the resources to be able to do that, or volunteering to try to help understand their problem and fix it if you believe that it's a skill issue. Reframing what you said in a more positive way could be: - I wonder what architecture they used for this and how it could be improved? - I wonder what resource limits they have that prevent them from handling this issue? - I wonder whats unique to a library that limits their ability to do what tech companies do? > — [Driver8 on hackernews](https://news.ycombinator.com/item?id=50395024) > What I still don't understand is who is doing it? I assume it's not the big companies like OpenAI or Google or Anthropic, right? So there are a bunch of small things out there doing the scraping and then they sell it to the big places? Or are these just smaller AI models doing the scraping and using it for themselves? Is there an underground ecosystem where people buy and sell this stuff? I understand why it's valuable to the AI companies I'm curious about how all this work, the economics of it all. > — [blakesterz on hackernews · 4 comments](https://news.ycombinator.com/item?id=50393275) > The problem is unintentional DOS due to traffic can constitute a felony under the CFAA; the case in question being Pulte Homes, Inc. v. Laborers' International Union 2011. After Pulte fired an employee represented by the union, LIUNA urged members to call and send email to the company, expressing their opinions. As a result of the increased traffic, the company's email system crashed. The Sixth Circuit ruled that the LIUNA's instruction to call and email "intentionally caused damage," reversing the lower court's decision. Now thankfully we have Sandvig v. Barr, 2020 where the DC Federal Court ruled that the CFAA does not criminalize the violation of a website's terms of service, but imo all bets are off in terms of interpretation of the law in the wild west of AI. > — [piltdownman on hackernews · 1 comments](https://news.ycombinator.com/item?id=50393693) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50392499) · 82 points · 48 comments - [hackernews](https://news.ycombinator.com/item?id=50386516) · 4 points · 0 comments ## Similar posts on daily.dev - [Publishers say no to AI scrapers, block bots at server level](https://daily.dev/posts/publishers-say-no-to-ai-scrapers-block-bots-at-server-level-nite7hp90) · The Register · 1 upvotes · 0 comments - [The web belongs to the machines now](https://daily.dev/posts/the-web-belongs-to-the-machines-now-oylvpdhmh) · TechCentral · 0 upvotes · 0 comments - [News Sites Are Blocking Internet Archive Over AI Scraping Fears](https://daily.dev/posts/news-sites-are-blocking-internet-archive-over-ai-scraping-fears-2mqqvfze1) · Hackaday · 0 upvotes · 0 comments - [Aggressive AI scrapers are making it kinda suck to run wikis](https://daily.dev/posts/aggressive-ai-scrapers-are-making-it-kinda-suck-to-run-wikis-n7we7rzew) · Lobsters · 1 upvotes · 0 comments - [An update on the scraper situation](https://daily.dev/posts/an-update-on-the-scraper-situation-f9grknbuz) · Hacker News · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#llm](https://daily.dev/tags/llm), [#crawling](https://daily.dev/tags/crawling) [View this post on daily.dev](https://daily.dev/posts/university-libraries-are-being-swarmed-by-ai-bots-putting-public-access-at-risk-ivxtowrzf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"University libraries are being swarmed by AI bots, putting public access at risk","url":"https://daily.dev/posts/university-libraries-are-being-swarmed-by-ai-bots-putting-public-access-at-risk-ivxtowrzf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/university-libraries-are-being-swarmed-by-ai-bots-putting-public-access-at-risk-ivxtowrzf"},"datePublished":"2026-11-06T16:14:17.286Z","dateModified":"2026-11-06T17:30:42.285Z","description":"AI scraping bots are overwhelming university library websites, sometimes knocking entire collections offline. The University of North Carolina at Chapel Hill...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1cbbe5073e43a1a94f9c371f3b153f6e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1cbbe5073e43a1a94f9c371f3b153f6e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/university-libraries-are-being-swarmed-by-ai-bots-putting-public-access-at-risk-ivxtowrzf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,llm,crawling","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"University libraries are being swarmed by AI bots, putting public access at risk"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/university-libraries-are-being-swarmed-by-ai-bots-putting-public-access-at-risk-ivxtowrzf#faq","mainEntity":[{"@type":"Question","name":"why can't libraries just block the IP addresses of AI scraping bots","acceptedAnswer":{"@type":"Answer","text":"Blocking individual IP addresses no longer works because a single company or bot farm can operate from thousands of different internet addresses simultaneously. Eric Hellman of Project Gutenberg notes this makes the old IP-blocking fix ineffective, forcing institutions toward other measures like CAPTCHAs or login requirements, which also end up blocking legitimate human users and compliant bots. Teams fighting scraper traffic compare mitigation approaches and outcomes on daily.dev."}},{"@type":"Question","name":"what did OpenAI, Google, Anthropic and Microsoft agree on for AI web scraping opt-outs","acceptedAnswer":{"@type":"Answer","text":"These companies expressed support for honoring a single, standardized opt-out signal placed in a website's robots.txt file that would block their web crawlers from harvesting data. However, compliance is voluntary, and the agreement does not cover problematic bots run by third-party companies or smaller AI startups, leaving a significant gap in protection. Developers tracking AI data-policy shifts follow standards like this one on daily.dev."}},{"@type":"Question","name":"how bad is AI bot traffic for library and archive websites","acceptedAnswer":{"@type":"Answer","text":"It is severe enough to periodically crash sites entirely. The University of North Carolina at Chapel Hill reported over 11,000 searches for Finnish music from thousands of distinct IP addresses in a single day, and AI scraping bots have been recorded requesting thousands of documents per second while downloading every available file format, unlike compliant crawlers that follow robots.txt. Infrastructure engineers sizing capacity against scraper surges can find similar case reports on daily.dev."}}]} ```