--- title: Breaking the open web won't stop the bad bots | daily.dev description: Two authors argue that proposed anti-bot laws like the federal Stealth Bot Prohibition Act and New York's Stealth Crawler Prohibition Act would harm legitimate... canonical: https://daily.dev/posts/breaking-the-open-web-won-t-stop-the-bad-bots-wbah6naxn twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Breaking the open web won't stop the bad bots | daily.dev og:description: Two authors argue that proposed anti-bot laws like the federal Stealth Bot Prohibition Act and New York's Stealth Crawler Prohibition Act would harm legitimate... og:url: https://daily.dev/posts/breaking-the-open-web-won-t-stop-the-bad-bots-wbah6naxn og:image: https://api.daily.dev/og/posts/wBaH6NAxn.png og:image:alt: Breaking the open web won't stop the bad bots og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Breaking the open web won't stop the bad bots **[Hacker News](https://daily.dev/sources/hn)** · 7 min read · 0 upvotes · 0 comments ## Summary Two authors argue that proposed anti-bot laws like the federal Stealth Bot Prohibition Act and New York's Stealth Crawler Prohibition Act would harm legitimate web scraping instead of stopping abusive AI crawlers. The federal bill lets the FTC fine stealth bot operators up to $53,000 for activity 'reasonably likely' to burden a website, without defining that standard. The New York bill lets publishers subpoena a crawler's identity without proving harm. The authors cite a WSJ investigation that analyzed 31,000 financial disclosures and the Internet Archive's Wayback Machine, which has archived over one trillion webpages, as examples of scraping used for public good. They argue liability should instead target conduct like overwhelming servers, fraud, or prohibited data uses. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What does the federal Stealth Bot Prohibition Act penalize and how much is the fine? The Stealth Bot Prohibition Act lets the FTC pursue penalties of up to $53,000 against anyone who uses a 'stealth bot', one that fails to disclose its identity and purpose, in a way 'reasonably likely to damage, impair, or burden' a website's technical or commercial operation. The bill provides no definition or standard for what counts as reasonably likely to impair a website. _Developers tracking crawler regulation and compliance risk can follow policy coverage like this on daily.dev._ ### How many webpages has the Internet Archive's Wayback Machine archived using crawlers? The Wayback Machine has used crawlers to archive more than one trillion webpages, creating a historical record of the internet that would otherwise constantly change or disappear. That archive has supported thousands of news stories and served as evidence in hundreds of court cases. _Anyone weighing the value of automated archiving against new anti-scraping rules can track this debate on daily.dev._ ### What rights does New York's Stealth Crawler Prohibition Act give publishers against anonymous crawlers? Under New York's Stealth Crawler Prohibition Act, which applies to covered news sites, a publisher can obtain a subpoena to unmask a crawler's identity without needing to show any harm. A company that discloses its identity but then accesses the site 10,000 times a minute would still comply with the law despite the cost it imposes. _Builders of crawlers and scraping tools can monitor state-level legal exposure like this via daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 39 comments across hackernews (as of 2026-10-30). **TL;DR:** Discussion is largely skeptical of the proposed laws, arguing identity disclosure doesn't address actual harm, jurisdictional limits make enforcement toothless against foreign bots, and the vague 'stealth bot' definitions could chill journalists and small scrapers while barely touching large or malicious actors. **Sentiment:** 10% positive · 25% mixed · 65% skeptical **The case for** - Some argue that requiring bots to identify themselves at least allows sites to block or rate-limit them. - A few note that if a company can be proven to be running harmful stealth bots, liability could extend to seizing foreign assets as a deterrent. **The pushback** - Knowing a crawler's identity doesn't indicate whether its behavior is actually harmful, since a compliant-but-abusive crawler could still overwhelm a server. - The laws only have jurisdiction over US entities, so bots from other countries can keep scraping unaffected. - Vague terms like 'reasonably likely to burden' a site are seen as unenforceable and likely to chill legitimate activity like journalism or research. - Distinguishing a 'bot' from an automated browser, ad blocker, or extension is seen as technically incoherent, risking overly broad bans. - Journalists or researchers without legal departments would be disadvantaged compared to large organizations able to seek exemptions. - Anti-bot measures like CAPTCHAs and Anubis-style challenges are criticized as burdensome to real users while being circumventable by well-resourced bots. **By community** - hackernews (skeptical): Commenters largely doubt the proposed bills would curb harmful bots and worry they'd instead burden legitimate scrapers, journalists, and small operators, with side debates on bot-detection schemes like Anubis and CAPTCHAs. **Hottest debate:** Whether requiring bots to self-identify is a meaningful solution or a hollow measure that only burdens legitimate actors while real bad actors evade jurisdiction entirely. **Open questions** - How would any of these laws be enforced against crawlers operating from outside the law's jurisdiction? - What technical or legal definition could distinguish a 'bot' from an automated browser, extension, or ad blocker? - Would journalists or small developers realistically be able to get exemptions the way large companies with legal departments could? **Highlights** > "Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not." It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill. > — [thereader12 on hackernews · 3 comments](https://news.ycombinator.com/item?id=50307532) > Even the premise seems to not understand what it's trying to require. There are zero humans accessing a website by sending electrical signals to Comcast using a telegraph or pulsing a flashlight into a strand of fiber. Every web browser is a computer program running on a computer. So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider? Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do? > — [AnthonyMouse on hackernews · 2 comments](https://news.ycombinator.com/item?id=50308494) > What gets me is how much the people who advocate for the "open web" have completely incorporated the values of Facebook and Google. Like there's that anime girl who harasses you when you visit so many open source web sites that just continues the pattern of being harassed by cookie dialogs and consent dialogs and pop-up ads and being asked for your email address and of course there is Wikipedia which is doing more and more of the same. Another web is possible but web populists don't really believe it. Just to get started, bots have been better at filling out CAPCHTAs than humans for a long time. I knew someone who got his site hacked by Russians circa 2008 because he published a neural network in freakin PHP that could read them. Anubis causes humans stress, slows down task completion, probably raises peoples blood pressure, and cumulatively causes a great deal of harm to users the same way Cloudflare's system does. Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc. CAPTCHAs, cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits with triple damages. > — [PaulHoule on hackernews · 3 comments](https://news.ycombinator.com/item?id=50307875) > If you have an endpoint with interesting information and you don't want it spammed with requests, require a proof of work from the client, scale up the hardness of it as the load goes up. Even a hard proof of work won't slow down somebody that makes one request per second because they are reading the site but will sighificantly interfere with crawlers that hit you 10000k a second currently at very little cost to them. The module doing proof of work might be a part of your frontend that is static and on CDN. If you want agents still have access to your data, just not through scraping, provide alternative format for them. Like periodic sqlite dump. > — [scotty79 on hackernews](https://news.ycombinator.com/item?id=50309374) > > In 2022, Tech Policy Press received a grant from the Omidyar Network for a research project evaluating vulnerabilities in the design, engineering, and use of encrypted messaging applications. https://omidyar.com/ is geofenced for me, presumably because they identified me as being anti open web. The article is ai lobby intended to repel a bill that would > allow the FTC to pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.” The definition of "stealth bot" is a bot that does not identify itself as such and their main grief to the proposal is > If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law. Which is the biggest load of shit I've heard given that all you need to do is identify yourself as a bot and respect robots.txt or be banned, the status quo before a bunch of assholes started systematically destroying the internet. > — [za\_creature on hackernews · 2 comments](https://news.ycombinator.com/item?id=50307510) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50263837) · 90 points · 39 comments ## Similar posts on daily.dev - [300 publishers take their fight against AI stealth bots to Congress](https://daily.dev/posts/300-publishers-take-their-fight-against-ai-stealth-bots-to-congress-9ux4sa3eq) · The Next Web · 1 upvotes · 0 comments - [An update on the scraper situation](https://daily.dev/posts/an-update-on-the-scraper-situation-f9grknbuz) · Hacker News · 0 upvotes · 0 comments - [Publishers say no to AI scrapers, block bots at server level](https://daily.dev/posts/publishers-say-no-to-ai-scrapers-block-bots-at-server-level-nite7hp90) · The Register · 1 upvotes · 0 comments - [Monitoring Legitimate Bot Traffic is Now a Cybersecurity Requirement](https://daily.dev/posts/monitoring-legitimate-bot-traffic-is-now-a-cybersecurity-requirement-nriqmyzgg) · Security Boulevard · 0 upvotes · 0 comments - [My Bot Got Banned for Being a Bot](https://daily.dev/posts/my-bot-got-banned-for-being-a-bot-8t7hvwobo) · Biilmann Blog · 1 upvotes · 0 comments --- Tags: [#crawling](https://daily.dev/tags/crawling), [#ai-governance](https://daily.dev/tags/ai-governance) [View this post on daily.dev](https://daily.dev/posts/breaking-the-open-web-won-t-stop-the-bad-bots-wbah6naxn) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Breaking the open web won't stop the bad bots","url":"https://daily.dev/posts/breaking-the-open-web-won-t-stop-the-bad-bots-wbah6naxn","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/breaking-the-open-web-won-t-stop-the-bad-bots-wbah6naxn"},"datePublished":"2026-10-30T18:36:11.968Z","dateModified":"2026-10-30T20:22:23.417Z","description":"Two authors argue that proposed anti-bot laws like the federal Stealth Bot Prohibition Act and New York's Stealth Crawler Prohibition Act would harm legitimate...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d8f60b77ca179109c21714f5b21852e1?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d8f60b77ca179109c21714f5b21852e1?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/breaking-the-open-web-won-t-stop-the-bad-bots-wbah6naxn","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"crawling,ai-governance","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Breaking the open web won't stop the bad bots"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/breaking-the-open-web-won-t-stop-the-bad-bots-wbah6naxn#faq","mainEntity":[{"@type":"Question","name":"What does the federal Stealth Bot Prohibition Act penalize and how much is the fine?","acceptedAnswer":{"@type":"Answer","text":"The Stealth Bot Prohibition Act lets the FTC pursue penalties of up to $53,000 against anyone who uses a 'stealth bot', one that fails to disclose its identity and purpose, in a way 'reasonably likely to damage, impair, or burden' a website's technical or commercial operation. The bill provides no definition or standard for what counts as reasonably likely to impair a website. Developers tracking crawler regulation and compliance risk can follow policy coverage like this on daily.dev."}},{"@type":"Question","name":"How many webpages has the Internet Archive's Wayback Machine archived using crawlers?","acceptedAnswer":{"@type":"Answer","text":"The Wayback Machine has used crawlers to archive more than one trillion webpages, creating a historical record of the internet that would otherwise constantly change or disappear. That archive has supported thousands of news stories and served as evidence in hundreds of court cases. Anyone weighing the value of automated archiving against new anti-scraping rules can track this debate on daily.dev."}},{"@type":"Question","name":"What rights does New York's Stealth Crawler Prohibition Act give publishers against anonymous crawlers?","acceptedAnswer":{"@type":"Answer","text":"Under New York's Stealth Crawler Prohibition Act, which applies to covered news sites, a publisher can obtain a subpoena to unmask a crawler's identity without needing to show any harm. A company that discloses its identity but then accesses the site 10,000 times a minute would still comply with the law despite the cost it imposes. Builders of crawlers and scraping tools can monitor state-level legal exposure like this via daily.dev."}}]} ```