--- title: Have it both ways: stay discoverable in search while... description: Cloudflare introduces a new 'Disallow AI Training' setting that lets site owners stay indexed in search while blocking mixed-use crawlers (a single crawler... canonical: https://daily.dev/posts/have-it-both-ways-stay-discoverable-in-search-while-disallowing-ai-training-dd2qofcki twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Have it both ways: stay discoverable in search while disallowing AI training | daily.dev og:description: Cloudflare introduces a new 'Disallow AI Training' setting that lets site owners stay indexed in search while blocking mixed-use crawlers (a single crawler... og:url: https://daily.dev/posts/have-it-both-ways-stay-discoverable-in-search-while-disallowing-ai-training-dd2qofcki og:image: https://api.daily.dev/og/posts/dD2qoFCki.png og:image:alt: Have it both ways: stay discoverable in search while disallowing AI training og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Have it both ways: stay discoverable in search while disallowing AI training **[Cloudflare](https://daily.dev/sources/cloudflare)** · 12 min read · 3 upvotes · 0 comments ## Summary Cloudflare introduces a new 'Disallow AI Training' setting that lets site owners stay indexed in search while blocking mixed-use crawlers (a single crawler serving both search and AI training) from using content for model training. Apple, Google, and Microsoft honor or have committed to honor this setting, and are designated 'Accountable' crawler operators alongside training-only crawlers from Amazon, Anthropic, Meta, and OpenAI. The 'Block AI Bots' legacy setting is deprecated in favor of granular Search, Training, and Agent controls; Managed Robots.txt is replaced by Bot Preference Sync. Existing customer settings migrate automatically based on prior configuration, with detailed mapping tables provided. New domains get preset configurations depending on whether the site is ad-monetized. Cloudflare also outlines plans to give site owners granular control over AI summary inclusion by early next year, and shares data on how AI summaries affect site traffic and conversion rates. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I stop AI training crawlers from using my site's content without losing Google, Bing, or Apple search indexing? Use Cloudflare's new Disallow AI Training setting, which publishes a Disallow directive in robots.txt that mixed-use crawlers like Googlebot, Bingbot, and Applebot honor for training while still allowing them to crawl for search. It also blocks training-only crawlers from Amazon, Anthropic, Meta, and OpenAI entirely, since blocking those does not affect search. _daily.dev surfaces updates like this for teams weighing search visibility against AI training exposure._ ### What happens to Cloudflare's old Block AI Bots setting after September 15, 2026? Block AI Bots is deprecated in favor of separate Search, Training, and Agent controls. Existing customers who used the legacy Block or Block on pages with ads setting are automatically migrated: Search becomes Allow, Training becomes Disallow AI Training, and Agent becomes Block on pages with ads, preserving prior practical behavior without requiring manual changes. _engineers tracking Cloudflare bot-management changes can follow migrations like this on daily.dev._ ### Does opting out of AI training with Google-Extended or Applebot-Extended hurt my search rankings? No, both Google and Apple state that disallowing their AI training crawlers (Google-Extended and Applebot-Extended) via robots.txt does not impact search ranking or a site's inclusion in search results. Bing is still building equivalent robots.txt-level support, targeted for early 2027, so NOARCHIVE tags remain the current workaround for Bing. _site owners deciding how to configure crawler permissions can track these vendor commitments on daily.dev._ ## Similar posts on daily.dev - [Your site, your rules: new AI traffic options for all customers](https://daily.dev/posts/your-site-your-rules-new-ai-traffic-options-for-all-customers-zrn0fx6z7) · Cloudflare · 0 upvotes · 0 comments - [Cloudflare gives AI crawlers a September deadline to pay up](https://daily.dev/posts/cloudflare-gives-ai-crawlers-a-september-deadline-to-pay-up-s5ncxffs7) · The Next Web · 0 upvotes · 0 comments - [Say it once: introducing Bot Preference SynC](https://daily.dev/posts/say-it-once-introducing-bot-preference-sync-chjkgyjqr) · Cloudflare · 1 upvotes · 0 comments - [Every AI Crawler in 2026: The Reference Table \(And Which Ones Your WAF Is Silently Blocking\)](https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn) · Security Boulevard · 1 upvotes · 1 comments --- Tags: [#seo](https://daily.dev/tags/seo), [#cloudflare](https://daily.dev/tags/cloudflare) [View this post on daily.dev](https://daily.dev/posts/have-it-both-ways-stay-discoverable-in-search-while-disallowing-ai-training-dd2qofcki) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Have it both ways: stay discoverable in search while disallowing AI training","url":"https://daily.dev/posts/have-it-both-ways-stay-discoverable-in-search-while-disallowing-ai-training-dd2qofcki","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/have-it-both-ways-stay-discoverable-in-search-while-disallowing-ai-training-dd2qofcki"},"datePublished":"2026-09-15T13:12:50.975Z","dateModified":"2026-09-17T05:18:16.622Z","description":"Cloudflare introduces a new 'Disallow AI Training' setting that lets site owners stay indexed in search while blocking mixed-use crawlers (a single crawler...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1744ccfd1abd522aea3cb7f3fbdc6510?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1744ccfd1abd522aea3cb7f3fbdc6510?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Cloudflare","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Cloudflare","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/38522e1d11354cd6b7af66f9d4316735","url":"https://daily.dev/sources/cloudflare"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/have-it-both-ways-stay-discoverable-in-search-while-disallowing-ai-training-dd2qofcki","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"seo,cloudflare","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Cloudflare","item":"https://daily.dev/sources/cloudflare"},{"@type":"ListItem","position":3,"name":"Have it both ways: stay discoverable in search while disallowing AI training"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/have-it-both-ways-stay-discoverable-in-search-while-disallowing-ai-training-dd2qofcki#faq","mainEntity":[{"@type":"Question","name":"How do I stop AI training crawlers from using my site's content without losing Google, Bing, or Apple search indexing?","acceptedAnswer":{"@type":"Answer","text":"Use Cloudflare's new Disallow AI Training setting, which publishes a Disallow directive in robots.txt that mixed-use crawlers like Googlebot, Bingbot, and Applebot honor for training while still allowing them to crawl for search. It also blocks training-only crawlers from Amazon, Anthropic, Meta, and OpenAI entirely, since blocking those does not affect search. daily.dev surfaces updates like this for teams weighing search visibility against AI training exposure."}},{"@type":"Question","name":"What happens to Cloudflare's old Block AI Bots setting after September 15, 2026?","acceptedAnswer":{"@type":"Answer","text":"Block AI Bots is deprecated in favor of separate Search, Training, and Agent controls. Existing customers who used the legacy Block or Block on pages with ads setting are automatically migrated: Search becomes Allow, Training becomes Disallow AI Training, and Agent becomes Block on pages with ads, preserving prior practical behavior without requiring manual changes. engineers tracking Cloudflare bot-management changes can follow migrations like this on daily.dev."}},{"@type":"Question","name":"Does opting out of AI training with Google-Extended or Applebot-Extended hurt my search rankings?","acceptedAnswer":{"@type":"Answer","text":"No, both Google and Apple state that disallowing their AI training crawlers (Google-Extended and Applebot-Extended) via robots.txt does not impact search ranking or a site's inclusion in search results. Bing is still building equivalent robots.txt-level support, targeted for early 2027, so NOARCHIVE tags remain the current workaround for Bing. site owners deciding how to configure crawler permissions can track these vendor commitments on daily.dev."}}]} ```