--- title: Cloudflare and OpenAI: making AI discovery better for... description: Cloudflare and OpenAI shared early results from a research pilot launched in July that uses signals from Cloudflare's network to help OpenAI's search crawler... canonical: https://daily.dev/posts/cloudflare-and-openai-making-ai-discovery-better-for-everyone-saiwhfn5a twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Cloudflare and OpenAI: making AI discovery better for everyone | daily.dev og:description: Cloudflare and OpenAI shared early results from a research pilot launched in July that uses signals from Cloudflare's network to help OpenAI's search crawler... og:url: https://daily.dev/posts/cloudflare-and-openai-making-ai-discovery-better-for-everyone-saiwhfn5a og:image: https://api.daily.dev/og/posts/sAiwhfn5a.png og:image:alt: Cloudflare and OpenAI: making AI discovery better for everyone og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Cloudflare and OpenAI: making AI discovery better for everyone **[Cloudflare](https://daily.dev/sources/cloudflare)** · 5 min read · 1 upvotes · 0 comments ## Summary Cloudflare and OpenAI shared early results from a research pilot launched in July that uses signals from Cloudflare's network to help OpenAI's search crawler discover and index new web pages more efficiently for ChatGPT. Using roughly 1% of Cloudflare's total network as a participating cohort, the pilot has helped OpenAI discover and index hundreds of millions of new web pages. The collaboration also tests signals to filter out nonexistent, redirected, or duplicate pages, flag compromised pages for safety, and detect when content has changed so crawlers avoid repeatedly fetching unchanged pages. Next steps include improving page selection, faster change signals, and measuring business outcomes after discovery. Cloudflare notes that signals about a page don't grant access permission, and the research uses metadata rather than page content for model training. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How many new web pages has the Cloudflare and OpenAI crawler discovery pilot indexed? The joint research pilot between Cloudflare and OpenAI has enabled OpenAI to discover and index hundreds of millions of new web pages. The pilot launched in July and used a participating cohort of about 1% of Cloudflare's total network, feeding signals to OpenAI's search crawler to help ChatGPT surface more useful, up-to-date sources to users. _Teams weighing how AI crawlers affect their site traffic can follow developments like this on daily.dev._ ### Does allowing an AI crawler to see metadata about my web pages mean it can access the page content for training? No, a signal about a page does not grant permission to access it. The Cloudflare-OpenAI pilot explicitly uses metadata about web pages for search and discovery purposes only, rather than sharing page content for model training, and domain owners' access choices remain in effect throughout the process. _Site owners deciding how much access to grant AI crawlers can track clarifications like this on daily.dev._ ### How can website owners control how AI crawlers like OpenAI's access their site? Website owners can use Cloudflare's AI Crawl Control to understand and manage how AI systems access their site. The pilot also uses filtering signals to keep crawlers away from nonexistent or redirected pages and change signals to tell crawlers when a page is worth revisiting, reducing unnecessary load from repeated crawling of unchanged content. _Developers managing crawler load and AI bot access can keep tabs on tools like this via daily.dev._ ## Similar posts on daily.dev - [Cloudflare and OpenAI pilot to make AI search fresher](https://daily.dev/posts/cloudflare-and-openai-pilot-to-make-ai-search-fresher-nodwmgakn) · The Next Web · 0 upvotes · 0 comments - [Making AI search smarter](https://daily.dev/posts/making-ai-search-smarter-vdumwudvn) · Cloudflare · 0 upvotes · 0 comments - [Cloudflare wants to build the economic layer of the AI web](https://daily.dev/posts/cloudflare-wants-to-build-the-economic-layer-of-the-ai-web-i5ywynw2y) · The New Stack · 0 upvotes · 0 comments - [Cloudflare’s new policy pushes AI companies to pay for publishers’ content](https://daily.dev/posts/cloudflare-s-new-policy-pushes-ai-companies-to-pay-for-publishers-content-er8fwfksu) · TechCrunch · 0 upvotes · 0 comments --- Tags: [#openai](https://daily.dev/tags/openai), [#seo](https://daily.dev/tags/seo), [#cloudflare](https://daily.dev/tags/cloudflare), [#crawling](https://daily.dev/tags/crawling), [#ai-search](https://daily.dev/tags/ai-search) [View this post on daily.dev](https://daily.dev/posts/cloudflare-and-openai-making-ai-discovery-better-for-everyone-saiwhfn5a) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Cloudflare and OpenAI: making AI discovery better for everyone","url":"https://daily.dev/posts/cloudflare-and-openai-making-ai-discovery-better-for-everyone-saiwhfn5a","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/cloudflare-and-openai-making-ai-discovery-better-for-everyone-saiwhfn5a"},"datePublished":"2026-10-15T13:01:53.844Z","dateModified":"2026-10-15T13:02:23.807Z","description":"Cloudflare and OpenAI shared early results from a research pilot launched in July that uses signals from Cloudflare's network to help OpenAI's search crawler...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a6be2a818dfaaa5a2555e4b44f0b6c86?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a6be2a818dfaaa5a2555e4b44f0b6c86?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Cloudflare","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Cloudflare","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/38522e1d11354cd6b7af66f9d4316735","url":"https://daily.dev/sources/cloudflare"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/cloudflare-and-openai-making-ai-discovery-better-for-everyone-saiwhfn5a","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"openai,seo,cloudflare,crawling,ai-search","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Cloudflare","item":"https://daily.dev/sources/cloudflare"},{"@type":"ListItem","position":3,"name":"Cloudflare and OpenAI: making AI discovery better for everyone"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/cloudflare-and-openai-making-ai-discovery-better-for-everyone-saiwhfn5a#faq","mainEntity":[{"@type":"Question","name":"How many new web pages has the Cloudflare and OpenAI crawler discovery pilot indexed?","acceptedAnswer":{"@type":"Answer","text":"The joint research pilot between Cloudflare and OpenAI has enabled OpenAI to discover and index hundreds of millions of new web pages. The pilot launched in July and used a participating cohort of about 1% of Cloudflare's total network, feeding signals to OpenAI's search crawler to help ChatGPT surface more useful, up-to-date sources to users. Teams weighing how AI crawlers affect their site traffic can follow developments like this on daily.dev."}},{"@type":"Question","name":"Does allowing an AI crawler to see metadata about my web pages mean it can access the page content for training?","acceptedAnswer":{"@type":"Answer","text":"No, a signal about a page does not grant permission to access it. The Cloudflare-OpenAI pilot explicitly uses metadata about web pages for search and discovery purposes only, rather than sharing page content for model training, and domain owners' access choices remain in effect throughout the process. Site owners deciding how much access to grant AI crawlers can track clarifications like this on daily.dev."}},{"@type":"Question","name":"How can website owners control how AI crawlers like OpenAI's access their site?","acceptedAnswer":{"@type":"Answer","text":"Website owners can use Cloudflare's AI Crawl Control to understand and manage how AI systems access their site. The pilot also uses filtering signals to keep crawlers away from nonexistent or redirected pages and change signals to tell crawlers when a page is worth revisiting, reducing unnecessary load from repeated crawling of unchanged content. Developers managing crawler load and AI bot access can keep tabs on tools like this via daily.dev."}}]} ```