<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/why-script-based-analytics-can-never-see-ai-crawlers-0hucznren" -->

---
title: Why script-based analytics can never see AI crawlers
description: Traditional analytics tools (GA4, Plausible, Fathom, Umami, PostHog) all measure pageviews via a client-side script, which means blocked scripts undercount...
canonical: https://daily.dev/posts/why-script-based-analytics-can-never-see-ai-crawlers-0hucznren
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Why script-based analytics can never see AI crawlers | daily.dev
og:description: Traditional analytics tools (GA4, Plausible, Fathom, Umami, PostHog) all measure pageviews via a client-side script, which means blocked scripts undercount...
og:url: https://daily.dev/posts/why-script-based-analytics-can-never-see-ai-crawlers-0hucznren
og:image: https://api.daily.dev/og/posts/0huCZnRen.png
og:image:alt: Why script-based analytics can never see AI crawlers
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Why script-based analytics can never see AI crawlers

**[Elvin](https://daily.dev/sources/p4aymb70h)** · [@elvinaghalarov](https://daily.dev/elvinaghalarov) · 5 min read · 1 upvotes · 0 comments

## Summary

Traditional analytics tools (GA4, Plausible, Fathom, Umami, PostHog) all measure pageviews via a client-side script, which means blocked scripts undercount real visitors and AI crawlers like GPTBot are entirely invisible since they never execute JavaScript. The proposed fix is moving measurement into the request path itself using a Cloudflare Worker route that counts pageviews as HTML is requested rather than when a script runs, making crawler traffic visible and immune to ad blockers. The piece details engineering constraints for building such an edge-based collector safely (never blocking the origin fetch, deferring measurement with waitUntil, touching only HTML documents, returning plain 502s on failure) and a subtle routing bug around Cloudflare's same-zone subrequest behavior that cannot be reproduced locally. It closes by promoting GhostPane, a self-hosted, cookieless analytics tool built on Workers, D1, and Durable Objects that implements this approach.

## Content

Every analytics product you have used works the same way: a `<script>` tag runs
  in the browser and sends a beacon. GA4, Plausible, Fathom, Umami, PostHog —
  different privacy stories, identical measurement point.

  That one design choice has two consequences none of them can engineer around:                                                                                                               

  - **A blocked or failed script is a pageview that never existed.** The missing
    share isn't small, and it isn't random — it's highest among exactly the
    technical audience most of us build for.
  - **Crawlers are completely invisible.** No crawler executes JavaScript. So no
    beacon fires, so no script-based product can tell you that GPTBot, ClaudeBot
    or PerplexityBot read your docs last night.

  That second one is the part people misread as a missing feature on a roadmap.
  It isn't. It's a property of *where* the measurement happens.

  ## Moving the measurement into the request path

  If you count the pageview when the HTML is *requested* instead of when a browser
  executes a script, both problems disappear at once. Nothing is asked of the
  client, so nothing can block it — and crawlers become ordinary requests you can                                                                                                             
  see.

  On Cloudflare that's a Worker route in front of the zone: forward the request to
  the origin, count it on the way past. The origin never knows. Your site can be
  on a VPS, Hetzner, Vercel, whatever — Cloudflare is already a reverse proxy in
  that path.
  
  The catch is that a *hosted* analytics SaaS can never do this. It doesn't run in
  your account, so it can't be in your request path. Self-hosting stops being an
  ideological preference here and starts being the thing that unlocks a capability.

  ## Writing code that sits between a stranger and someone else's website                                                                                                                     

  This is the part I found genuinely interesting to build, because the engineering
  standard is different from normal app code. Rules I ended up with:

  **1. Nothing is awaited before the origin fetch.** Not a KV read, not a site
  lookup, not a hash. The first thing that happens to a request is being sent on.                                                                                                             

  **2. All measurement happens after the response is returned**, under
  `waitUntil`. A cold Durable Object or a database outage must not be able to add
  a millisecond to a visitor's page load — or fail it.

  **3. The app router is never reached.** The fork lives in the Worker's `fetch`,                                                                                                             
  before Hono exists. Every middleware I wrote was written for *my* pages: the
  canonical-host redirect would 301 someone else's visitors to my dashboard, and
  the header hardener would staple my CSP and HSTS onto their responses.

  **4. Only HTML documents get touched.** Redirects, asset streams and WebSocket                                                                                                              
  upgrades come back byte-identical.

  **5. If the origin fetch fails, return a plain 502** — not a styled error page.
  Whatever is broken is between the visitor and the site; a branded page would
  only misdirect whoever is debugging it.

  ## The bug you cannot reproduce locally

  Here's the one that cost me time. Cloudflare sends a Worker's subrequest to the
  **origin** when the URL matches a route belonging to that same Worker. That rule
  is exactly what makes the pass-through a pass-through instead of an infinite
  loop.
  
  It also cannot be exercised on your machine — a local `fetch` to the same
  hostname goes straight back into the Worker. So a misconfigured route serves the
  *wrong site* without ever throwing an error. Local tests can prove your
  decisions (what gets counted, what gets de-duplicated, what's left alone); only
  a real request against a real zone proves your routing.

  The verification is deliberately boring:

  ```bash
  curl -I https://example.com/
  # x-gp-edge: 1

  One header on HTML documents. That's the only thing the whole feature changes
  about a page.

  The number you get out of it

  Run both collectors — request path and script — and tag each observation with
  its source. One page load produces two observations with the same visitor hash
  and path, so they de-duplicate into one pageview, and you get:

  ┌─────────┬────────────────────────────────────────────────┐
  │ counter │                    meaning                     │
  ├─────────┼────────────────────────────────────────────────┤
  │ edge    │ document pageviews seen in the request path    │
  ├─────────┼────────────────────────────────────────────────┤
  │ both    │ ...of those, the ones the script also reported │
  ├─────────┼────────────────────────────────────────────────┤
  │ tag     │ pageviews only the script reported             │
  └─────────┴────────────────────────────────────────────────┘

  edge - both is your coverage gap. Not an industry average for ad-blocker
  usage — a number you measured on your own traffic.

  And tag being non-zero is expected, not a discrepancy: a SPA's client-side
  route changes make no HTTP request at all, so the script is the only thing that
  can see them. Same for engagement time, scroll depth, Web Vitals and custom
  events. The two collectors are complements, not alternatives.
  
  What it actually costs

  A route on example.com/* makes every HTTP request to that zone a Worker
  invocation — images and stylesheets included, because routes run ahead of the
  cache. That's the real price of edge mode, and it's why a non-document request
  has to be a bare pass-through with zero work done on it.
  
  ---

  I've been building this into GhostPane — cookieless
  analytics you host in your own Cloudflare account (Workers + D1 + Durable
  Objects), which is the only reason the request-path trick is available at all.

  Curious whether anyone has measured their own script-coverage gap this way. If                                                                                                              
  you have, what did the number look like?

  Two things before you post:

  - **The GhostPane link is a placeholder** (`https://github.com/`) — swap in your storefront or repo URL.                                                                                    
  - **Tags:** `cloudflare`, `webdev`, `analytics`, `serverless`, `privacy`. The title is the main lever on daily.dev; if you want it less absolute, "Your analytics can't see AI crawlers — and that's
  structural, not a missing feature" also works.

## Similar posts on daily.dev

- [Do ad blockers block Plausible Analytics?](https://daily.dev/posts/do-ad-blockers-block-plausible-analytics--89ngldmum) · Plausible Analytics · 0 upvotes · 0 comments
- [Your analytics are lying to you](https://daily.dev/posts/your-analytics-are-lying-to-you-tinikpi9m) · Lobsters · 0 upvotes · 0 comments
- [Which AI actually reads your site? Two months of LLM traffic, measured—Martian Chronicles, Evil Martians’ team blog](https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv) · Martian Chronicles · 1 upvotes · 0 comments
- [Every Ad On My Network Was Invisible To Agents](https://daily.dev/posts/every-ad-on-my-network-was-invisible-to-agents-g5ircxc1a) · API Evangelist · 0 upvotes · 0 comments
- [The Developer’s Way to Track Traffic: Self-Hosting Ackee or Plausible](https://daily.dev/posts/the-developer-s-way-to-track-traffic-self-hosting-ackee-or-plausible-89bg31qfb) · Ruby Flow · 0 upvotes · 0 comments

---

Tags: [#cloudflare](https://daily.dev/tags/cloudflare), [#self-hosting](https://daily.dev/tags/self-hosting), [#edge-computing](https://daily.dev/tags/edge-computing)

[View this post on daily.dev](https://daily.dev/posts/why-script-based-analytics-can-never-see-ai-crawlers-0hucznren)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/why-script-based-analytics-can-never-see-ai-crawlers-0hucznren","headline":"Why script-based analytics can never see AI crawlers","text":"Traditional analytics tools (GA4, Plausible, Fathom, Umami, PostHog) all measure pageviews via a client-side script, which means blocked scripts undercount real visitors and AI crawlers like GPTBot are entirely invisible since they never execute JavaScript. The proposed fix is moving measurement into the request path itself using a Cloudflare Worker route that counts pageviews as HTML is requested rather than when a script runs, making crawler traffic visible and immune to ad blockers. The piece details engineering constraints for building such an edge-based collector safely (never blocking the origin fetch, deferring measurement with waitUntil, touching only HTML documents, returning plain 502s on failure) and a subtle routing bug around Cloudflare's same-zone subrequest behavior that cannot be reproduced locally. It closes by promoting GhostPane, a self-hosted, cookieless analytics tool built on Workers, D1, and Durable Objects that implements this approach.","url":"https://daily.dev/posts/why-script-based-analytics-can-never-see-ai-crawlers-0hucznren","datePublished":"2026-08-18T08:09:23.156Z","dateModified":"2026-09-13T20:47:34.178Z","author":{"@type":"Person","name":"Elvin","url":"https://daily.dev/elvinaghalarov","image":"https://avatars.githubusercontent.com/u/64695589?v=4","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":60}},"image":"https://media.daily.dev/image/upload/s--nM2_iQDB--/f_auto/v1787040563/posts/0huCZnRen?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/p4aymb70h","name":"Elvin"}}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Elvin","item":"https://daily.dev/sources/p4aymb70h"},{"@type":"ListItem","position":3,"name":"Why script-based analytics can never see AI crawlers"}]}
```

