--- title: Every AI Crawler in 2026: The Reference Table (And Which... description: AI crawlers in 2026 fall into three distinct classes: training (feeds model weights), retrieval (feeds live AI search citations), and user-triggered (fetches... canonical: https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Every AI Crawler in 2026: The Reference Table (And Which Ones Your WAF Is Silently Blocking) | daily.dev og:description: AI crawlers in 2026 fall into three distinct classes: training (feeds model weights), retrieval (feeds live AI search citations), and user-triggered (fetches... og:url: https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn og:image: https://api.daily.dev/og/posts/UJFhBfIVN.png og:image:alt: Every AI Crawler in 2026: The Reference Table (And Which Ones Your WAF Is Silently Blocking) og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Every AI Crawler in 2026: The Reference Table (And Which Ones Your WAF Is Silently Blocking) **[Security Boulevard](https://daily.dev/sources/securityboulevard)** · 12 min read · 1 upvotes · 1 comments ## Summary AI crawlers in 2026 fall into three distinct classes: training (feeds model weights), retrieval (feeds live AI search citations), and user-triggered (fetches pages on human request). Blocking them indiscriminately — especially via WAF rules that predate AI search — can silently remove a site from AI-generated answers without anyone in marketing knowing. Cloudflare data shows AI crawlers account for ~27% of verified bot traffic. A reference table covers ~18 bots (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, etc.) with their class, robots.txt token, and blocking cost. Key insight: blocking GPTBot (training) does not remove you from ChatGPT search, which uses OAI-SearchBot. On September 15, 2026, Cloudflare changes defaults so Training and Agent crawlers are blocked by default for new domains on ad-displaying pages, while Search crawlers remain allowed. A sample robots.txt template and four verification steps (origin logs, external robots.txt fetch, response code checks, IP range verification) are provided. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Which AI crawlers should I never block if I want to appear in AI search answers? Googlebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot should never be blocked. These are retrieval crawlers that build the indexes AI search products query at answer time, so blocking any of them removes a site from that product's cited answers. Googlebot is the most consequential because it also gates eligibility for Google AI Overviews and AI Mode. _Keep pace with which crawlers gate AI search visibility by following daily.dev coverage of AI infrastructure changes._ ### Does blocking GPTBot remove my site from ChatGPT search results? No, blocking GPTBot does not remove a site from ChatGPT search results. GPTBot only collects training data; ChatGPT's search feature uses OAI-SearchBot instead, and live page reads during a conversation use ChatGPT-User. A site can block GPTBot for training purposes and still appear in ChatGPT search citations as long as OAI-SearchBot and ChatGPT-User remain allowed. _Developers weighing which bots to allow can track crawler policy nuances like this on daily.dev._ ### What changes to AI crawler defaults is Cloudflare making on September 15, 2026? Cloudflare is introducing three AI traffic categories - Search, Agent, and Training - each with its own default. For new domains, Training and Agent crawlers get blocked by default on pages that display ads, while Search crawlers remain allowed by default. Existing domains keep their current configuration, so the risk falls on newly onboarded properties inheriting a stricter posture than an established site. _Teams auditing bot-management defaults before a platform change can follow updates like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@rizzdev** · 0 upvotes > The WAF rules the piece wants you to audit only expose one AI category. Training retrieval and user triggered stay collapsed into the same toggle. Origin logs then record the decision without showing which class triggered it so the per class allow list stays untestable in practice ## Similar posts on daily.dev - [Publishers say no to AI scrapers, block bots at server level](https://daily.dev/posts/publishers-say-no-to-ai-scrapers-block-bots-at-server-level-nite7hp90) · The Register · 1 upvotes · 0 comments - [Your site, your rules: new AI traffic options for all customers](https://daily.dev/posts/your-site-your-rules-new-ai-traffic-options-for-all-customers-zrn0fx6z7) · Cloudflare · 0 upvotes · 0 comments --- Tags: [#security](https://daily.dev/tags/security), [#seo](https://daily.dev/tags/seo), [#cloudflare](https://daily.dev/tags/cloudflare) [View this post on daily.dev](https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Every AI Crawler in 2026: The Reference Table (And Which Ones Your WAF Is Silently Blocking)","url":"https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn"},"datePublished":"2026-07-29T05:30:31.257Z","dateModified":"2026-09-14T06:53:20.444Z","description":"AI crawlers in 2026 fall into three distinct classes: training (feeds model weights), retrieval (feeds live AI search citations), and user-triggered (fetches...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8fe9eeb967de205fd6743e2535f16178?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8fe9eeb967de205fd6743e2535f16178?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Security Boulevard","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Security Boulevard","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/3613c832180040de8d85bb29f74395be","url":"https://daily.dev/sources/securityboulevard"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"security,seo,cloudflare","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Security Boulevard","item":"https://daily.dev/sources/securityboulevard"},{"@type":"ListItem","position":3,"name":"Every AI Crawler in 2026: The Reference Table (And Which Ones Your WAF Is Silently Blocking)"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn","comment":[{"@type":"Comment","text":"The WAF rules the piece wants you to audit only expose one AI category. Training retrieval and user triggered stay collapsed into the same toggle. Origin logs then record the decision without showing which class triggered it so the per class allow list stays untestable in practice","datePublished":"2026-07-31T03:34:51.993Z","url":"https://daily.dev/posts/UJFhBfIVN#c-iRJghzPlG","author":{"@type":"Person","name":"Andrew","url":"https://daily.dev/rizzdev","image":"https://media.daily.dev/image/upload/s--35vRfXAA--/f_auto/v1785413727/avatars/avatar_fQ7ttCKtPuu6tDvcgQXv5?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn#faq","mainEntity":[{"@type":"Question","name":"Which AI crawlers should I never block if I want to appear in AI search answers?","acceptedAnswer":{"@type":"Answer","text":"Googlebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot should never be blocked. These are retrieval crawlers that build the indexes AI search products query at answer time, so blocking any of them removes a site from that product's cited answers. Googlebot is the most consequential because it also gates eligibility for Google AI Overviews and AI Mode. Keep pace with which crawlers gate AI search visibility by following daily.dev coverage of AI infrastructure changes."}},{"@type":"Question","name":"Does blocking GPTBot remove my site from ChatGPT search results?","acceptedAnswer":{"@type":"Answer","text":"No, blocking GPTBot does not remove a site from ChatGPT search results. GPTBot only collects training data; ChatGPT's search feature uses OAI-SearchBot instead, and live page reads during a conversation use ChatGPT-User. A site can block GPTBot for training purposes and still appear in ChatGPT search citations as long as OAI-SearchBot and ChatGPT-User remain allowed. Developers weighing which bots to allow can track crawler policy nuances like this on daily.dev."}},{"@type":"Question","name":"What changes to AI crawler defaults is Cloudflare making on September 15, 2026?","acceptedAnswer":{"@type":"Answer","text":"Cloudflare is introducing three AI traffic categories - Search, Agent, and Training - each with its own default. For new domains, Training and Agent crawlers get blocked by default on pages that display ads, while Search crawlers remain allowed by default. Existing domains keep their current configuration, so the risk falls on newly onboarded properties inheriting a stricter posture than an established site. Teams auditing bot-management defaults before a platform change can follow updates like this on daily.dev."}}]} ```