--- title: Someone is running mass vulnerability scans, spoofing AI... description: A continuously updated dashboard analyzes bot versus human traffic patterns across 5,000+ websites, breaking down AI scraping, fetching, search indexing, and... canonical: https://daily.dev/posts/someone-is-running-mass-vulnerability-scans-spoofing-ai-bots-like-claudebot-homewd2zq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot | daily.dev og:description: A continuously updated dashboard analyzes bot versus human traffic patterns across 5,000+ websites, breaking down AI scraping, fetching, search indexing, and... og:url: https://daily.dev/posts/someone-is-running-mass-vulnerability-scans-spoofing-ai-bots-like-claudebot-homewd2zq og:image: https://api.daily.dev/og/posts/HOmewd2zq.png og:image:alt: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot **[Hacker News](https://daily.dev/sources/hn)** · 21 min read · 0 upvotes · 0 comments ## Summary A continuously updated dashboard analyzes bot versus human traffic patterns across 5,000+ websites, breaking down AI scraping, fetching, search indexing, and browsing activity by agent type. It ranks the most active bots (Bingbot, Googlebot, AhrefsBot lead overall traffic), the top AI data scrapers training LLMs (ClaudeBot, meta-externalagent, Amazonbot, GPTBot, Bytespider), AI assistants fetching content for answers (ChatGPT-User dominates at 86.4%), AI search crawlers providing citations (PetalBot, Amzn-SearchBot, Applebot), and autonomous browsing agents (Google-Agent, Manus-User, ChatGPT Agent). It also measures robots.txt compliance effectiveness, showing which bots best respect disallow rules and which ignore them most. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Which AI bot does the most web scraping for training AI models? ClaudeBot leads AI data scraping activity, accounting for roughly 27% of scraping traffic among AI data scrapers, followed by meta-externalagent at about 19.5% and Amazonbot at around 19.2%. GPTBot and Bytespider trail behind at roughly 9.4% and 7.3% respectively. These bots download website content to build datasets used for training large language models. _Track which AI scrapers are pulling your content by following AI bot traffic trends on daily.dev._ ### Which AI assistant bot fetches the most website content for chat answers? ChatGPT-User accounts for the overwhelming majority of AI assistant fetching activity, at roughly 86.4% of that traffic category, far ahead of DuckAssistBot (3.3%), Perplexity-User (2.7%), and Claude-User (2.5%). These bots retrieve page content in real time to include in AI-generated answers to user prompts. _Developers weighing which AI assistants to optimize content for can follow bot traffic data on daily.dev._ ### How well do AI bots like GPTBot and ClaudeBot respect robots.txt disallow rules? Compliance varies significantly by bot: GPTBot shows around 24.6% continued traffic after being disallowed, CCBot about 22.6%, and ClaudeBot roughly 21.7%, meaning a meaningful share of these bots keep crawling despite robots.txt rules blocking them. In contrast, bots like LinkupBot, serpstatbot, and UptimeRobot show 100% effectiveness, essentially stopping traffic entirely once disallowed. _Site owners deciding how to configure robots.txt against AI crawlers can track compliance patterns on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 37 comments across hackernews (as of 2026-08-12). **TL;DR:** Commenters largely agree that spoofed AI bot user-agents are just typical background internet noise rather than a genuinely new phenomenon, with debate over whether the reported surge and its scale claims are actually significant or just self-promotion. **Sentiment:** 10% positive · 30% mixed · 60% skeptical **The case for** - Some found the observation useful, noting the surge in spoofing patterns across many unrelated sites and that many targeted paths relate to newish AI coding tools, suggesting exposed AI tooling or leaked secrets are being probed. - One person shared a self-built open-source honeypot/blocklist as a practical tool for tracking this kind of traffic. **The pushback** - Several argued this is nothing new — mass scanning and fake user-agents have existed for years and are just business as usual for any exposed server. - Some questioned whether thousands of hits across thousands of sites is actually large-scale or noteworthy, calling it easy for any random actor to trigger. - One commenter accused the frequent poster of being the researcher/company behind the data and dismissed their repeated defenses as untrustworthy cheerleading. - Attribution of scan origin (country/ASN/IP) was called into question as technically shaky given how internet routing and packet injection work. - Blocking based on VPS/ASN or country IP ranges was discussed as only partially effective, since many legitimate users also come from non-residential IPs. **By community** - hackernews (heated): A back-and-forth between the story's promoter and skeptics dominates, with skeptics downplaying the novelty and scale of the findings and one accusing the promoter of self-interested cheerleading, alongside tangential technical debates about IP attribution and bot-blocking strategies. **Hottest debate:** Whether the surge in spoofed bot traffic is a genuinely novel, significant finding or just routine internet background noise being oversold. **Open questions** - What specific new vulnerability or exploit these scans might be searching for. - Whether the person repeatedly defending and promoting the findings is affiliated with the company behind the dashboard. - How to reliably distinguish VPS-originated traffic from legitimate residential/commercial users at the network level. **Highlights** > Is this your company? If it is, your cheerleading makes you very hard to trust. If it’s not, I’m sure that everyone gets the point - you adore everything about this research and can’t see any possible problems. > — [hluska on hackernews · 1 comments](https://news.ycombinator.com/item?id=49274821) > It's still not really anything special. Thousands isn't even large scale. Any random bozo can trigger that. > — [KomoD on hackernews · 1 comments](https://news.ycombinator.com/item?id=49274148) > I run an open source honeypot that collects these botnet scans and produces blocklists. Blocklist download and configuration: https://knock-knock.net/blocklist Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net API: http://knock-knock.net/api > — [djkurlander on hackernews](https://news.ycombinator.com/item?id=49274117) > Someone is always running mass vulnerability scans. That's a "water is wet" state of the Internet. > — [pjc50 on hackernews · 2 comments](https://news.ycombinator.com/item?id=49273270) > People or their agents must be accidentally committing or publishing their repository level secrets and configs with enough regularity that it’s worth scanning. > — [thedougd on hackernews · 1 comments](https://news.ycombinator.com/item?id=49274387) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49272569) · 80 points · 37 comments ## Similar posts on daily.dev - [Your site, your rules: new AI traffic options for all customers](https://daily.dev/posts/your-site-your-rules-new-ai-traffic-options-for-all-customers-zrn0fx6z7) · Cloudflare · 0 upvotes · 0 comments - [Dead internet theory becomes measurable fact as AI agents flood the web](https://daily.dev/posts/dead-internet-theory-becomes-measurable-fact-as-ai-agents-flood-the-web-4uwxc0w1l) · The Next Web · 1 upvotes · 0 comments - [Publishers say no to AI scrapers, block bots at server level](https://daily.dev/posts/publishers-say-no-to-ai-scrapers-block-bots-at-server-level-nite7hp90) · The Register · 1 upvotes · 0 comments - [The web belongs to the machines now](https://daily.dev/posts/the-web-belongs-to-the-machines-now-oylvpdhmh) · TechCentral · 0 upvotes · 0 comments --- Tags: [#seo](https://daily.dev/tags/seo) [View this post on daily.dev](https://daily.dev/posts/someone-is-running-mass-vulnerability-scans-spoofing-ai-bots-like-claudebot-homewd2zq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot","url":"https://daily.dev/posts/someone-is-running-mass-vulnerability-scans-spoofing-ai-bots-like-claudebot-homewd2zq","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/someone-is-running-mass-vulnerability-scans-spoofing-ai-bots-like-claudebot-homewd2zq"},"datePublished":"2026-08-12T15:20:11.248Z","dateModified":"2026-08-12T16:36:39.658Z","description":"A continuously updated dashboard analyzes bot versus human traffic patterns across 5,000+ websites, breaking down AI scraping, fetching, search indexing, and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8578c38557181ce238dca76ee3d2dd4e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8578c38557181ce238dca76ee3d2dd4e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/someone-is-running-mass-vulnerability-scans-spoofing-ai-bots-like-claudebot-homewd2zq","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"seo","timeRequired":"PT21M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/someone-is-running-mass-vulnerability-scans-spoofing-ai-bots-like-claudebot-homewd2zq#faq","mainEntity":[{"@type":"Question","name":"Which AI bot does the most web scraping for training AI models?","acceptedAnswer":{"@type":"Answer","text":"ClaudeBot leads AI data scraping activity, accounting for roughly 27% of scraping traffic among AI data scrapers, followed by meta-externalagent at about 19.5% and Amazonbot at around 19.2%. GPTBot and Bytespider trail behind at roughly 9.4% and 7.3% respectively. These bots download website content to build datasets used for training large language models. Track which AI scrapers are pulling your content by following AI bot traffic trends on daily.dev."}},{"@type":"Question","name":"Which AI assistant bot fetches the most website content for chat answers?","acceptedAnswer":{"@type":"Answer","text":"ChatGPT-User accounts for the overwhelming majority of AI assistant fetching activity, at roughly 86.4% of that traffic category, far ahead of DuckAssistBot (3.3%), Perplexity-User (2.7%), and Claude-User (2.5%). These bots retrieve page content in real time to include in AI-generated answers to user prompts. Developers weighing which AI assistants to optimize content for can follow bot traffic data on daily.dev."}},{"@type":"Question","name":"How well do AI bots like GPTBot and ClaudeBot respect robots.txt disallow rules?","acceptedAnswer":{"@type":"Answer","text":"Compliance varies significantly by bot: GPTBot shows around 24.6% continued traffic after being disallowed, CCBot about 22.6%, and ClaudeBot roughly 21.7%, meaning a meaningful share of these bots keep crawling despite robots.txt rules blocking them. In contrast, bots like LinkupBot, serpstatbot, and UptimeRobot show 100% effectiveness, essentially stopping traffic entirely once disallowed. Site owners deciding how to configure robots.txt against AI crawlers can track compliance patterns on daily.dev."}}]} ```