--- title: Why Google, Common Crawl, and ChatGPT See Different... description: A hands-on experiment fetches the same MDN caching page through six different observation layers — raw HTTP, Playwright-rendered DOM, Googlebot's Rich Results... canonical: https://daily.dev/posts/why-google-common-crawl-and-chatgpt-see-different-versions-of-the-same-url-ms4dv96mt twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why Google, Common Crawl, and ChatGPT See Different Versions of the Same URL | daily.dev og:description: A hands-on experiment fetches the same MDN caching page through six different observation layers — raw HTTP, Playwright-rendered DOM, Googlebot's Rich Results... og:url: https://daily.dev/posts/why-google-common-crawl-and-chatgpt-see-different-versions-of-the-same-url-ms4dv96mt og:image: https://api.daily.dev/og/posts/ms4dv96Mt.png og:image:alt: Why Google, Common Crawl, and ChatGPT See Different Versions of the Same URL og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why Google, Common Crawl, and ChatGPT See Different Versions of the Same URL **[JavaScript in Plain English](https://daily.dev/sources/jsPlainEnglish)** · 21 min read · 0 upvotes · 0 comments ## Summary A hands-on experiment fetches the same MDN caching page through six different observation layers — raw HTTP, Playwright-rendered DOM, Googlebot's Rich Results Test render, a Common Crawl WARC record, a Wayback Machine snapshot, and three LLMs with tool use — and finds each returns meaningfully different content. Highlights: the Playwright DOM lost 37 code samples because MDN moves them into shadow-DOM custom elements; Googlebot's renderer flattens shadow DOM and recovers the samples but bloats the payload ~20% and breaks grep-ability; Common Crawl served a stale edge-cached build despite being captured after the latest edit; Wayback added ~12KB of injected scripts; and gemini-3.6-flash silently returned only a third of the page while reporting it as complete. The takeaway is that RAG and scraping pipelines must match the observation layer to the actual task, and verify tool-using LLM output against a real acquisition layer rather than trusting confident-sounding but potentially truncated answers. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does a Playwright-rendered DOM sometimes contain fewer code samples than the raw HTML response for the same page? Some sites, like MDN, move server-rendered code samples into custom elements whose actual content lives inside a shadow root, invisible to the light DOM. A default Playwright extraction using page.content() or innerText will not see this content unless the script explicitly walks each element's shadowRoot to pull the text out, so naive scraping can silently drop dozens of code blocks that raw HTTP fetches or cached crawls still contain. _Teams building scraping or RAG pipelines can compare notes on shadow-DOM extraction pitfalls on daily.dev._ ### Is Google's Rich Results Test a good way to get page content for a RAG pipeline? No, it should only be used as a diagnostic to audit your own extractor rather than as an ingestion source. On a tested MDN page, the Web Rendering Service output was about 20% larger than the raw HTTP body due to hydration debris like Lit comment markers and copy-button markup, and it split code samples into syntax-highlighting spans that break plain substring search, even though it does flatten shadow DOM content that a raw Playwright extraction misses. _Anyone weighing acquisition layers for scraping or RAG ingestion can find comparisons like this on daily.dev._ ### Can I trust an LLM's tool-using web fetch to report on an entire webpage? Not without verification: in one test, gemini-3.6-flash fetched only about the first third of an MDN caching article, missing 24 of 34 headings and the entire see-also section, yet reported the partial content with the same confident tone as if it had read the whole page, with no indication the fetch was truncated. Other models like gpt-5.6-sol and sonnet-5 performed accurately on the same task, correctly capturing nearly all headings and code samples. _Developers relying on AI tools to summarize live pages can track these accuracy gaps via daily.dev._ ## Similar posts on daily.dev - [Crawl entire websites with a single API call using Browser Rendering](https://daily.dev/posts/crawl-entire-websites-with-a-single-api-call-using-browser-rendering-3zv6nlrlk) · Hacker News · 0 upvotes · 0 comments - [Scraping websites and bot-protection](https://daily.dev/posts/scraping-websites-and-bot-protection-fnv9adnrq) · A Java geek · 7 upvotes · 0 comments - [Behind the scenes of an AI-Driven Web Scraping System](https://daily.dev/posts/behind-the-scenes-of-an-ai-driven-web-scraping-system-apsqb0lnn) · RUBYLAND · 0 upvotes · 0 comments - [My local LLM's real problem wasn't hallucination, it was a two-year blind spot](https://daily.dev/posts/my-local-llm-s-real-problem-wasn-t-hallucination-it-was-a-two-year-blind-spot-eqscg8rag) · XDA Developers · 0 upvotes · 0 comments --- Tags: [#html](https://daily.dev/tags/html), [#rag](https://daily.dev/tags/rag), [#crawling](https://daily.dev/tags/crawling) [View this post on daily.dev](https://daily.dev/posts/why-google-common-crawl-and-chatgpt-see-different-versions-of-the-same-url-ms4dv96mt) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why Google, Common Crawl, and ChatGPT See Different Versions of the Same URL","url":"https://daily.dev/posts/why-google-common-crawl-and-chatgpt-see-different-versions-of-the-same-url-ms4dv96mt","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-google-common-crawl-and-chatgpt-see-different-versions-of-the-same-url-ms4dv96mt"},"datePublished":"2026-08-16T19:29:33.322Z","dateModified":"2026-09-14T07:05:24.926Z","description":"A hands-on experiment fetches the same MDN caching page through six different observation layers — raw HTTP, Playwright-rendered DOM, Googlebot's Rich Results...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2fb5b621ee3b744ccc9cc194afb9f2cf?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2fb5b621ee3b744ccc9cc194afb9f2cf?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"JavaScript in Plain English","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"JavaScript in Plain English","logo":"https://media.daily.dev/image/upload/s--Dg5QTRuI--/f_auto,q_auto/v1686665538/logos/pe","url":"https://daily.dev/sources/jsPlainEnglish"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-google-common-crawl-and-chatgpt-see-different-versions-of-the-same-url-ms4dv96mt","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"html,rag,crawling","timeRequired":"PT21M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"JavaScript in Plain English","item":"https://daily.dev/sources/jsPlainEnglish"},{"@type":"ListItem","position":3,"name":"Why Google, Common Crawl, and ChatGPT See Different Versions of the Same URL"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-google-common-crawl-and-chatgpt-see-different-versions-of-the-same-url-ms4dv96mt#faq","mainEntity":[{"@type":"Question","name":"Why does a Playwright-rendered DOM sometimes contain fewer code samples than the raw HTML response for the same page?","acceptedAnswer":{"@type":"Answer","text":"Some sites, like MDN, move server-rendered code samples into custom elements whose actual content lives inside a shadow root, invisible to the light DOM. A default Playwright extraction using page.content() or innerText will not see this content unless the script explicitly walks each element's shadowRoot to pull the text out, so naive scraping can silently drop dozens of code blocks that raw HTTP fetches or cached crawls still contain. Teams building scraping or RAG pipelines can compare notes on shadow-DOM extraction pitfalls on daily.dev."}},{"@type":"Question","name":"Is Google's Rich Results Test a good way to get page content for a RAG pipeline?","acceptedAnswer":{"@type":"Answer","text":"No, it should only be used as a diagnostic to audit your own extractor rather than as an ingestion source. On a tested MDN page, the Web Rendering Service output was about 20% larger than the raw HTTP body due to hydration debris like Lit comment markers and copy-button markup, and it split code samples into syntax-highlighting spans that break plain substring search, even though it does flatten shadow DOM content that a raw Playwright extraction misses. Anyone weighing acquisition layers for scraping or RAG ingestion can find comparisons like this on daily.dev."}},{"@type":"Question","name":"Can I trust an LLM's tool-using web fetch to report on an entire webpage?","acceptedAnswer":{"@type":"Answer","text":"Not without verification: in one test, gemini-3.6-flash fetched only about the first third of an MDN caching article, missing 24 of 34 headings and the entire see-also section, yet reported the partial content with the same confident tone as if it had read the whole page, with no indication the fetch was truncated. Other models like gpt-5.6-sol and sonnet-5 performed accurately on the same task, correctly capturing nearly all headings and code samples. Developers relying on AI tools to summarize live pages can track these accuracy gaps via daily.dev."}}]} ```