--- title: Which AI actually reads your site? Two months of LLM... description: Evil Martians instrumented their own site with server-side middleware to track LLM agent traffic over two months, capturing ~268,000 agent requests vs ~107,000... canonical: https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Which AI actually reads your site? Two months of LLM traffic, measured—Martian Chronicles, Evil Martians’ team blog | daily.dev og:description: Evil Martians instrumented their own site with server-side middleware to track LLM agent traffic over two months, capturing ~268,000 agent requests vs ~107,000... og:url: https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv og:image: https://api.daily.dev/og/posts/baplAa6Bv.png og:image:alt: Which AI actually reads your site? Two months of LLM traffic, measured—Martian Chronicles, Evil Martians’ team blog og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Which AI actually reads your site? Two months of LLM traffic, measured—Martian Chronicles, Evil Martians’ team blog **[Martian Chronicles](https://daily.dev/sources/evilmartians)** · 15 min read · 1 upvotes · 0 comments ## Summary Evil Martians instrumented their own site with server-side middleware to track LLM agent traffic over two months, capturing ~268,000 agent requests vs ~107,000 human pageviews. Key findings: ChatGPT-User accounts for 73% of all agent traffic and reads rendered HTML almost exclusively (0.1% Markdown); Claude Code requests Markdown 76% of the time via HTTP content negotiation. The llms.txt file received ~770 direct fetches but only 37 came from named AI assistants — the rest were SEO bots and crawlers. A hidden AI hint div earned zero attributable fetches. The practical takeaway: keep HTML clean for ChatGPT, implement Accept: text/markdown content negotiation for coding agents, and measure your own logs rather than following generic GEO advice. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Does ChatGPT actually fetch the Markdown version of a website when Accept: text/markdown content negotiation is offered? No, ChatGPT-User essentially never fetches Markdown. Out of nearly 197,000 requests measured over two months, only 272 were for the Markdown version (0.1%), despite ChatGPT-User accounting for 73% of all agent traffic to the site. It reads rendered HTML almost exclusively, so optimizing HTML output matters more than building a Markdown pipeline for this client. _Anyone deciding where to invest GEO effort can compare agent behavior patterns like these on daily.dev._ ### Do AI assistants actually read and use llms.txt files? Not meaningfully. Out of roughly 770 direct fetches of /llms.txt and /llms-full.txt over two months, only 37 came from named AI assistants like ChatGPT, Claude, Perplexity, GPTBot, or OAI-SearchBot; the rest were search crawlers, SEO bots, and scanner tools. Referrer traffic that looked like link-following was 106 of 117 hits from a single bot spoofing an old Chrome version, not a real AI agent. _Developers weighing whether llms.txt is worth their time can dig into measurement writeups like this on daily.dev._ ### How much of Claude Code's traffic to a website requests Markdown instead of HTML via content negotiation? Claude Code requests the Markdown version 76% of the time, using the Accept: text/markdown HTTP header rather than any site-specific convention. Of roughly 23,300 requests attributed to Claude over two months, 17,814 were Markdown fetches, making Claude Code alone responsible for 43% of all legitimate Markdown traffic to the site after filtering out anonymous indexer bots. _Teams building content negotiation for coding agents can track real numbers like these on daily.dev._ ## Similar posts on daily.dev - [Making your site visible to LLMs: 6 techniques that work, 8 that don't—Martian Chronicles, Evil Martians’ team blog](https://daily.dev/posts/making-your-site-visible-to-llms-6-techniques-that-work-8-that-don-t-martian-chronicles-evil-mart-qfl0kutgl) · RUBYLAND · 2 upvotes · 0 comments - [Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot](https://daily.dev/posts/someone-is-running-mass-vulnerability-scans-spoofing-ai-bots-like-claudebot-homewd2zq) · Hacker News · 0 upvotes · 0 comments - [Your next user is an agent, not a person](https://daily.dev/posts/your-next-user-is-an-agent-not-a-person-ye7z4yogy) · SD Times · 0 upvotes · 0 comments - [Dead internet theory becomes measurable fact as AI agents flood the web](https://daily.dev/posts/dead-internet-theory-becomes-measurable-fact-as-ai-agents-flood-the-web-4uwxc0w1l) · The Next Web · 1 upvotes · 0 comments --- [View this post on daily.dev](https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Which AI actually reads your site? Two months of LLM traffic, measured—Martian Chronicles, Evil Martians’ team blog","url":"https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv"},"datePublished":"2026-07-21T13:59:22.896Z","dateModified":"2026-09-14T06:13:59.064Z","description":"Evil Martians instrumented their own site with server-side middleware to track LLM agent traffic over two months, capturing ~268,000 agent requests vs ~107,000...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/49032065b5d9b8c9659717a5ca42b5e2?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/49032065b5d9b8c9659717a5ca42b5e2?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Martian Chronicles","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Martian Chronicles","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/6334c9de96a7414ebbf4b7667ba572c5","url":"https://daily.dev/sources/evilmartians"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"","timeRequired":"PT15M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Martian Chronicles","item":"https://daily.dev/sources/evilmartians"},{"@type":"ListItem","position":3,"name":"Which AI actually reads your site? Two months of LLM traffic, measured—Martian Chronicles, Evil Martians’ team blog"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured-martian-chronicles-evil-mart-baplaa6bv#faq","mainEntity":[{"@type":"Question","name":"Does ChatGPT actually fetch the Markdown version of a website when Accept: text/markdown content negotiation is offered?","acceptedAnswer":{"@type":"Answer","text":"No, ChatGPT-User essentially never fetches Markdown. Out of nearly 197,000 requests measured over two months, only 272 were for the Markdown version (0.1%), despite ChatGPT-User accounting for 73% of all agent traffic to the site. It reads rendered HTML almost exclusively, so optimizing HTML output matters more than building a Markdown pipeline for this client. Anyone deciding where to invest GEO effort can compare agent behavior patterns like these on daily.dev."}},{"@type":"Question","name":"Do AI assistants actually read and use llms.txt files?","acceptedAnswer":{"@type":"Answer","text":"Not meaningfully. Out of roughly 770 direct fetches of /llms.txt and /llms-full.txt over two months, only 37 came from named AI assistants like ChatGPT, Claude, Perplexity, GPTBot, or OAI-SearchBot; the rest were search crawlers, SEO bots, and scanner tools. Referrer traffic that looked like link-following was 106 of 117 hits from a single bot spoofing an old Chrome version, not a real AI agent. Developers weighing whether llms.txt is worth their time can dig into measurement writeups like this on daily.dev."}},{"@type":"Question","name":"How much of Claude Code's traffic to a website requests Markdown instead of HTML via content negotiation?","acceptedAnswer":{"@type":"Answer","text":"Claude Code requests the Markdown version 76% of the time, using the Accept: text/markdown HTTP header rather than any site-specific convention. Of roughly 23,300 requests attributed to Claude over two months, 17,814 were Markdown fetches, making Claude Code alone responsible for 43% of all legitimate Markdown traffic to the site after filtering out anonymous indexer bots. Teams building content negotiation for coding agents can track real numbers like these on daily.dev."}}]} ```