--- title: Three directories failed to read our docs page. I blamed... description: A firewall rule ordering bug, not client-side rendering, was blocking crawlers and API importers from reading a docs page. A WAF block rule for non-browser... canonical: https://daily.dev/posts/three-directories-failed-to-read-our-docs-page-i-blamed-client-side-rendering-it-was-our-own-firew-i2qcvksx0 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Three directories failed to read our docs page. I blamed client-side rendering. It was our own firewall. | daily.dev og:description: A firewall rule ordering bug, not client-side rendering, was blocking crawlers and API importers from reading a docs page. A WAF block rule for non-browser... og:url: https://daily.dev/posts/three-directories-failed-to-read-our-docs-page-i-blamed-client-side-rendering-it-was-our-own-firew-i2qcvksx0 og:image: https://api.daily.dev/og/posts/I2qCvKSx0.png og:image:alt: Three directories failed to read our docs page. I blamed client-side rendering. It was our own firewall. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Three directories failed to read our docs page. I blamed client-side rendering. It was our own firewall. **[AlphaAI](https://daily.dev/sources/alphai)** · [@mikhailmakeev](https://daily.dev/mikhailmakeev) · 2 min read · 0 upvotes · 0 comments ## Summary A firewall rule ordering bug, not client-side rendering, was blocking crawlers and API importers from reading a docs page. A WAF block rule for non-browser user agents (curl allowed, python-requests/Go-http-client/HeadlessChrome blocked) sat at a higher priority than the allow rule for bots, and since Block terminates rule evaluation, the allow never triggered. A second, quieter bug: the allow rule only listed page paths, not /_next/static/, so non-browser clients got the HTML document but hit a CAPTCHA on every stylesheet request, producing screenshots with no CSS. The fix was swapping the priority numbers on two existing rules. GPTBot and Perplexity were unaffected the whole time; it was unnamed importers and directories getting bounced. ## Content A directory tried to import our docs page and produced a screenshot with no CSS. Two others got fragments. I spent a day on the theory that the page renders client-side and naive fetchers see an empty shell. One curl killed that: 200, and 31,034 characters of visible text. curl was also the reason I'd been wrong for a day. Our WAF blocks a list of user agents and curl isn't on it, so the client I reached for to prove the page was fine was the one client allowed through. ``` curl 200 python-requests 403 Go-http-client 403 HeadlessChrome 403 ``` Nothing in the block rule was incorrect. It sat at priority 9. The rule that lets bots read our marketing pages sat at 10. A WAF walks rules in priority order and Block terminates, so the allow never got a turn. The fix was swapping two numbers. Neither rule body changed. The quieter half took longer to see. That allow rule listed the pages themselves and not /_next/static/. A non-browser client got the document, then a CAPTCHA on every stylesheet. Which is the screenshot with no CSS, exactly. 6,990 of those in 23 days from 1,340 distinct IPs, and I'd never once looked at a 405 on a static asset. The thing I was actually afraid of turned out fine: GPTBot and Perplexity had been passing all along, 499 hits in a day and every one a 200. It was the unnamed importers that were bouncing. Second time this same block list has cost us something. The first time it sat there eating robots.txt and sitemap.xml for weeks. I diagnosed that one with curl too. The page all of this was about: https://alphai.io/developers (free key, 20 a minute, no card) ## Similar posts on daily.dev - [Every AI Crawler in 2026: The Reference Table \(And Which Ones Your WAF Is Silently Blocking\)](https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn) · Security Boulevard · 1 upvotes · 1 comments - [How Cloudflare Bot Fight Mode Quietly Killed Our Google Rankings](https://daily.dev/posts/how-cloudflare-bot-fight-mode-quietly-killed-our-google-rankings-lbdhmurms) · The Infinity · 8 upvotes · 2 comments - [Blocking LLM crawlers, without JavaScript](https://daily.dev/posts/blocking-llm-crawlers-without-javascript-5rpssvlpe) · Lobsters · 4 upvotes · 0 comments --- Tags: [#security](https://daily.dev/tags/security), [#curl](https://daily.dev/tags/curl) [View this post on daily.dev](https://daily.dev/posts/three-directories-failed-to-read-our-docs-page-i-blamed-client-side-rendering-it-was-our-own-firew-i2qcvksx0) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/three-directories-failed-to-read-our-docs-page-i-blamed-client-side-rendering-it-was-our-own-firew-i2qcvksx0","headline":"Three directories failed to read our docs page. I blamed client-side rendering. It was our own firewall.","text":"A firewall rule ordering bug, not client-side rendering, was blocking crawlers and API importers from reading a docs page. A WAF block rule for non-browser user agents (curl allowed, python-requests/Go-http-client/HeadlessChrome blocked) sat at a higher priority than the allow rule for bots, and since Block terminates rule evaluation, the allow never triggered. A second, quieter bug: the allow rule only listed page paths, not /_next/static/, so non-browser clients got the HTML document but hit a CAPTCHA on every stylesheet request, producing screenshots with no CSS. The fix was swapping the priority numbers on two existing rules. GPTBot and Perplexity were unaffected the whole time; it was unnamed importers and directories getting bounced.","url":"https://daily.dev/posts/three-directories-failed-to-read-our-docs-page-i-blamed-client-side-rendering-it-was-our-own-firew-i2qcvksx0","datePublished":"2026-09-03T07:25:37.219Z","dateModified":"2026-09-03T07:25:57.781Z","author":{"@type":"Person","name":"Mikhail Makeev","url":"https://daily.dev/mikhailmakeev","image":"https://lh3.googleusercontent.com/a/ACg8ocIjcvDU1gmJ9-KJDeTYqY2rMdl5QxcWYtLTGSXrj8ubF4nwar_w=s96-c","description":"Owner of alphai.io","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":1490}},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/squads/alphai","name":"AlphaAI"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AlphaAI","item":"https://daily.dev/squads/alphai"},{"@type":"ListItem","position":3,"name":"Three directories failed to read our docs page. I blamed client-side rendering. It was our own firewall."}]} ```