--- title: Every Ad On My Network Was Invisible To Agents | daily.dev description: An API-focused publisher discovers that every ad on his own self-hosted network was invisible to AI agents because the ad embed relies on client-side... canonical: https://daily.dev/posts/every-ad-on-my-network-was-invisible-to-agents-g5ircxc1a twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Every Ad On My Network Was Invisible To Agents | daily.dev og:description: An API-focused publisher discovers that every ad on his own self-hosted network was invisible to AI agents because the ad embed relies on client-side... og:url: https://daily.dev/posts/every-ad-on-my-network-was-invisible-to-agents-g5ircxc1a og:image: https://api.daily.dev/og/posts/G5iRCxC1a.png og:image:alt: Every Ad On My Network Was Invisible To Agents og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Every Ad On My Network Was Invisible To Agents **[API Evangelist](https://daily.dev/sources/apievangelist)** · 9 min read · 0 upvotes · 0 comments ## Summary An API-focused publisher discovers that every ad on his own self-hosted network was invisible to AI agents because the ad embed relies on client-side JavaScript that crawlers never execute. He rejects putting inventory in llms.txt (Ahrefs found 97% of llms.txt files get zero requests) and rejects Time's approach of cloaking different content to agents via CDN edge detection, calling it a governance risk. Instead he builds a same-URL, no-cloaking markdown twin for every page (5,184 posts, plus 26,721 structured documents for APIs.io), served via content negotiation with a Vary header, containing zero ads in phase one to measure real agent fetch rates before ever inserting inventory, with strict future caps (15% byte limit, disclosure per chunk, no third-party ads) enforced in the build rather than policy. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why don't AI agents see ads that are inserted into a webpage with JavaScript? AI crawlers like ClaudeBot fetch the raw HTML response and never execute JavaScript, so any ad embed that relies on a script filling in a div at runtime returns byte-for-byte identical HTML with an empty ad slot. A test comparing ClaudeBot's fetch to a browser's fetch of the same 72,808-byte page showed the ad container present but empty in both, since the ad network never renders server-side. _Publishers rethinking how ads and content reach agents can track these shifts on daily.dev._ ### Do AI agents actually fetch llms.txt files from websites? Rarely. Ahrefs examined server logs and found that ninety-seven percent of llms.txt files received zero requests in May 2026, despite roughly thirty-six thousand of them existing across the web. Agents overwhelmingly fetch the actual page rather than the dedicated discovery file, making llms.txt an unreliable channel for surfacing content or inventory to AI crawlers. _Developers weighing llms.txt against other agent-discovery approaches can follow this debate on daily.dev._ ### Is serving different content to AI crawlers than to human visitors considered cloaking? Yes, serving materially different content at the same URL depending on whether a human, an AI agent, or a search engine crawler is requesting it matches the traditional definition of cloaking, a practice Google has penalized with delisting for two decades. One publisher was found serving markdown to AI crawlers and HTML to humans, with advertiser content making up forty-one to seventy percent of the bytes an agent ingests on some pages. _Teams weighing agent-content strategies against cloaking risk can track precedent on daily.dev._ ## Similar posts on daily.dev - [How Time Serves Ads to AI Agents](https://daily.dev/posts/how-time-serves-ads-to-ai-agents-5mhfmc0bf) · API Evangelist · 2 upvotes · 0 comments - [The 49MB Web Page](https://daily.dev/posts/the-49mb-web-page-c2modzgaa) · Hacker News · 16 upvotes · 2 comments - [Making your site visible to LLMs: 6 techniques that work, 8 that don't—Martian Chronicles, Evil Martians’ team blog](https://daily.dev/posts/making-your-site-visible-to-llms-6-techniques-that-work-8-that-don-t-martian-chronicles-evil-mart-qfl0kutgl) · RUBYLAND · 2 upvotes · 0 comments - [Which AI actually reads your site? Two months of LLM traffic, measured—Martian Chronicles, Evil Martians’ team blog](https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv) · Martian Chronicles · 1 upvotes · 0 comments - [The AI Licensing Market Is Visible in HTTP Status Codes](https://daily.dev/posts/the-ai-licensing-market-is-visible-in-http-status-codes-uuinrpghn) · API Evangelist · 0 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/every-ad-on-my-network-was-invisible-to-agents-g5ircxc1a) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Every Ad On My Network Was Invisible To Agents","url":"https://daily.dev/posts/every-ad-on-my-network-was-invisible-to-agents-g5ircxc1a","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/every-ad-on-my-network-was-invisible-to-agents-g5ircxc1a"},"datePublished":"2026-08-13T10:21:07.972Z","dateModified":"2026-08-13T10:21:34.191Z","description":"An API-focused publisher discovers that every ad on his own self-hosted network was invisible to AI agents because the ad embed relies on client-side...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c839015f6b57af8dedb6e91c1300a6da?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c839015f6b57af8dedb6e91c1300a6da?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"API Evangelist","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"API Evangelist","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/a36244ae67bc4f41a605f780267ecb5f","url":"https://daily.dev/sources/apievangelist"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/every-ad-on-my-network-was-invisible-to-agents-g5ircxc1a","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"API Evangelist","item":"https://daily.dev/sources/apievangelist"},{"@type":"ListItem","position":3,"name":"Every Ad On My Network Was Invisible To Agents"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/every-ad-on-my-network-was-invisible-to-agents-g5ircxc1a#faq","mainEntity":[{"@type":"Question","name":"Why don't AI agents see ads that are inserted into a webpage with JavaScript?","acceptedAnswer":{"@type":"Answer","text":"AI crawlers like ClaudeBot fetch the raw HTML response and never execute JavaScript, so any ad embed that relies on a script filling in a div at runtime returns byte-for-byte identical HTML with an empty ad slot. A test comparing ClaudeBot's fetch to a browser's fetch of the same 72,808-byte page showed the ad container present but empty in both, since the ad network never renders server-side. Publishers rethinking how ads and content reach agents can track these shifts on daily.dev."}},{"@type":"Question","name":"Do AI agents actually fetch llms.txt files from websites?","acceptedAnswer":{"@type":"Answer","text":"Rarely. Ahrefs examined server logs and found that ninety-seven percent of llms.txt files received zero requests in May 2026, despite roughly thirty-six thousand of them existing across the web. Agents overwhelmingly fetch the actual page rather than the dedicated discovery file, making llms.txt an unreliable channel for surfacing content or inventory to AI crawlers. Developers weighing llms.txt against other agent-discovery approaches can follow this debate on daily.dev."}},{"@type":"Question","name":"Is serving different content to AI crawlers than to human visitors considered cloaking?","acceptedAnswer":{"@type":"Answer","text":"Yes, serving materially different content at the same URL depending on whether a human, an AI agent, or a search engine crawler is requesting it matches the traditional definition of cloaking, a practice Google has penalized with delisting for two decades. One publisher was found serving markdown to AI crawlers and HTML to humans, with advertiser content making up forty-one to seventy percent of the bytes an agent ingests on some pages. Teams weighing agent-content strategies against cloaking risk can track precedent on daily.dev."}}]} ```