--- title: Grok is obsessed with our docs. Just not the parts we... description: An engineer at Handsontable describes discovering that xAI's Grok crawler hammered their documentation site with 4.5 million edge requests and 87,000 page... canonical: https://daily.dev/posts/grok-is-obsessed-with-our-docs-just-not-the-parts-we-built-for-it--mwn8cklfw twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Grok is obsessed with our docs. Just not the parts we built for it. | daily.dev og:description: An engineer at Handsontable describes discovering that xAI's Grok crawler hammered their documentation site with 4.5 million edge requests and 87,000 page... og:url: https://daily.dev/posts/grok-is-obsessed-with-our-docs-just-not-the-parts-we-built-for-it--mwn8cklfw og:image: https://api.daily.dev/og/posts/mWN8CklFw.png og:image:alt: Grok is obsessed with our docs. Just not the parts we built for it. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Grok is obsessed with our docs. Just not the parts we built for it. **[Joseph Petty](https://daily.dev/sources/zpb9mu4ubnjor54jkhxoc)** · [@greenflux](https://daily.dev/greenflux) · 1 min read · 0 upvotes · 0 comments ## Summary An engineer at Handsontable describes discovering that xAI's Grok crawler hammered their documentation site with 4.5 million edge requests and 87,000 page renders within four hours, all originating from xAI's Colossus data center in Memphis. After catching the spike, the team redirected the crawler traffic to their MCP (Model Context Protocol) endpoint instead of the rendered docs, then spent a week analyzing logs to understand how AI crawlers behave, what content they prioritize, and what they skip - noting Grok ignored the AI-specific documentation they had built for exactly this purpose. ## Content I built an AI assistant for our docs at [**Handsontable**](https://www.linkedin.com/company/handsontable/), and then Grok found it. 4.5 million edge requests and 87,000 page renders in only 4 hours, all from xAI's Colossus data center in Memphis. I caught it during the scan and pushed a fix to redirect traffic to our MCP, then spent a week analyzing the logs and learning how crawlers work, what they look for, and what they ignore. [https://handsontable.com/blog/grok-is-obsessed-with-our-docs](https://handsontable.com/blog/grok-is-obsessed-with-our-docs) ## Similar posts on daily.dev - [xAI Opens Grok Build 0.1 to Developers via API](https://daily.dev/posts/xai-opens-grok-build-0-1-to-developers-via-api-fz592bcvp) · DevOps.com · 1 upvotes · 0 comments - [xAI Releases Grok Skills and Updates Tool Calling Responses API](https://daily.dev/posts/xai-releases-grok-skills-and-updates-tool-calling-responses-api-70xrc56pd) · InfoQ · 0 upvotes · 0 comments - [The Quiet Arrival of Grok Build 0.1 in a Wild Week for the xAI Empire](https://daily.dev/posts/the-quiet-arrival-of-grok-build-0-1-in-a-wild-week-for-the-xai-empire-fus2j39ne) · Kilo Blog · 0 upvotes · 0 comments - [I can't believe how good this AI coding model is, and it isn't from OpenAI or Anthropic](https://daily.dev/posts/i-can-t-believe-how-good-this-ai-coding-model-is-and-it-isn-t-from-openai-or-anthropic-kiu1haqe7) · XDA Developers · 0 upvotes · 0 comments --- Tags: [#mcp](https://daily.dev/tags/mcp), [#grok](https://daily.dev/tags/grok) [View this post on daily.dev](https://daily.dev/posts/grok-is-obsessed-with-our-docs-just-not-the-parts-we-built-for-it--mwn8cklfw) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/grok-is-obsessed-with-our-docs-just-not-the-parts-we-built-for-it--mwn8cklfw","headline":"Grok is obsessed with our docs. Just not the parts we built for it.","text":"An engineer at Handsontable describes discovering that xAI's Grok crawler hammered their documentation site with 4.5 million edge requests and 87,000 page renders within four hours, all originating from xAI's Colossus data center in Memphis. After catching the spike, the team redirected the crawler traffic to their MCP (Model Context Protocol) endpoint instead of the rendered docs, then spent a week analyzing logs to understand how AI crawlers behave, what content they prioritize, and what they skip - noting Grok ignored the AI-specific documentation they had built for exactly this purpose.","url":"https://daily.dev/posts/grok-is-obsessed-with-our-docs-just-not-the-parts-we-built-for-it--mwn8cklfw","datePublished":"2026-09-29T15:24:38.989Z","dateModified":"2026-09-29T19:19:59.057Z","author":{"@type":"Person","name":"Joseph Petty","url":"https://daily.dev/greenflux","image":"https://media.daily.dev/image/upload/s--veWd0iRh--/f_auto/v1725792042/avatars/avatar_ZPb9mU4UbNJoR54JkhxoC","description":"Head of DX @ Handsontable\nFounder/ Freelancer @ GreenFlux, LLC greenflux.github.io","worksFor":{"@type":"Organization","name":"Appsmith","logo":"https://res.cloudinary.com/daily-now/image/upload/s--UoClaMwT--/f_auto/v1725651404/companies/appsmith"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":2120}},"image":"https://media.daily.dev/image/upload/s--QTF6xWvH--/f_auto/v1790695485/posts/mWN8CklFw?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/zpb9mu4ubnjor54jkhxoc","name":"Joseph Petty"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joseph Petty","item":"https://daily.dev/sources/zpb9mu4ubnjor54jkhxoc"},{"@type":"ListItem","position":3,"name":"Grok is obsessed with our docs. Just not the parts we built for it."}]} ```