--- title: OpenAI, Google, Microsoft and Anthropic will honor one... description: Website operators will soon be able to opt out of AI training once instead of configuring each crawler separately. OpenAI, Google, Microsoft and Anthropic have... canonical: https://daily.dev/posts/openai-google-microsoft-and-anthropic-will-honor-one-robots-txt-signal-for-ai-training-opt-outs-srjmtpgd1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: OpenAI, Google, Microsoft and Anthropic will honor one robots.txt signal for AI training opt-outs | daily.dev og:description: Website operators will soon be able to opt out of AI training once instead of configuring each crawler separately. OpenAI, Google, Microsoft and Anthropic have... og:url: https://daily.dev/posts/openai-google-microsoft-and-anthropic-will-honor-one-robots-txt-signal-for-ai-training-opt-outs-srjmtpgd1 og:image: https://api.daily.dev/og/posts/SRjmTpGD1.png og:image:alt: OpenAI, Google, Microsoft and Anthropic will honor one robots.txt signal for AI training opt-outs og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # OpenAI, Google, Microsoft and Anthropic will honor one robots.txt signal for AI training opt-outs **[Collections](https://daily.dev/sources/collections)** · 2 min read · 1 upvotes · 0 comments ## Summary Website operators will soon be able to opt out of AI training once instead of configuring each crawler separately. OpenAI, Google, Microsoft and Anthropic have said they will honor a single shared robots.txt signal. Sites add a new policy token, `__usage_ai-training`, listing paths to exclude. Anthropic's Kevin Kelley wrote the draft and submitted it to the IETF as an individual submission, so it has no formal standards status. Microsoft starts February 1, 2027, replacing its NoArchive and NoCache tags, while Google targets early 2027. The signal is only a preference and can be overridden by private agreements; Meta voiced support but made no commitment. ## Content Four of the biggest AI companies will respect one robots.txt signal that lets a website say it doesn't want its content used for AI training. Until now, operators had to identify each provider's crawler and write separate rules for each one. ## How it works A site adds a group to robots.txt using a new policy token, `__usage_ai-training`, and lists the paths it wants excluded. One entry covers every participating provider. The draft comes from Anthropic's Kevin Kelley, submitted to the IETF as an individual submission. It has no formal standards status. It also draws on the ongoing work of the IETF AI Preferences working group. ## Who supports it and when - **Anthropic** already supports it. - **OpenAI** plans to honor it within the coming months and says implementation details will follow. Its existing crawler controls stay in effect. - **Microsoft** starts on February 1, 2027, and the signal will replace its NoArchive and NoCache tags. - **Google** is targeting early 2027. OpenAI is also encouraging other AI providers to adopt the same signal. Meta's communications head voiced support but didn't commit to honoring it. ## Limits The signal states a preference. It isn't a crawling order, and private agreements between a site and a provider can override it. Still, getting four competitors to agree on one mechanism is a real improvement over a patchwork of per-crawler rules. Whether it holds up depends on Meta and the smaller providers, and on how closely the big four follow through once their dates arrive. ## Questions this post answers ### How do I block my website's content from being used in AI training with robots.txt? Add a group to robots.txt using the new policy token `__usage_ai-training` and list the paths you want excluded. OpenAI, Google, Microsoft and Anthropic have said they will honor this single shared signal instead of requiring separate rules per crawler. The draft was written by Anthropic's Kevin Kelley and submitted to the IETF as an individual submission, so it currently has no formal standards status. _Site operators weighing AI crawler policies can follow standards coverage like this on daily.dev._ ### When will Google and Microsoft start honoring the AI training opt-out signal in robots.txt? Microsoft starts on February 1, 2027, where the new signal replaces its existing NoArchive and NoCache tags. Google is targeting early 2027. OpenAI plans to support the signal in the coming months and will publish implementation details later, while Anthropic has already said it will honor it. _Teams planning crawler and SEO configs around this rollout can track the timeline on daily.dev._ ### Is the robots.txt AI training opt-out signal legally binding on crawlers? No, the signal states only a preference, not a crawling order. Private agreements between a site and an AI company can override it. The proposal also has no formal standards status since it was submitted to the IETF as an individual submission rather than through a working group process. _Developers deciding how much to trust crawler opt-outs can weigh the limits via daily.dev._ ## Similar posts on daily.dev - [Say it once: introducing Bot Preference SynC](https://daily.dev/posts/say-it-once-introducing-bot-preference-sync-chjkgyjqr) · Cloudflare · 1 upvotes · 0 comments - [Publishers say no to AI scrapers, block bots at server level](https://daily.dev/posts/publishers-say-no-to-ai-scrapers-block-bots-at-server-level-nite7hp90) · The Register · 1 upvotes · 0 comments - [robots.txt in the age of AI crawlers](https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee) · Flavio Copes · 1 upvotes · 1 comments - [Amazonbot is finally respecting robots.txt](https://daily.dev/posts/amazonbot-is-finally-respecting-robots-txt-g5kabkdrl) · Hacker News · 12 upvotes · 3 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#openai](https://daily.dev/tags/openai), [#anthropic](https://daily.dev/tags/anthropic) [View this post on daily.dev](https://daily.dev/posts/openai-google-microsoft-and-anthropic-will-honor-one-robots-txt-signal-for-ai-training-opt-outs-srjmtpgd1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"OpenAI, Google, Microsoft and Anthropic will honor one robots.txt signal for AI training opt-outs","url":"https://daily.dev/posts/openai-google-microsoft-and-anthropic-will-honor-one-robots-txt-signal-for-ai-training-opt-outs-srjmtpgd1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/openai-google-microsoft-and-anthropic-will-honor-one-robots-txt-signal-for-ai-training-opt-outs-srjmtpgd1"},"datePublished":"2026-10-31T12:30:53.693Z","dateModified":"2026-10-31T13:58:45.675Z","description":"Website operators will soon be able to opt out of AI training once instead of configuring each crawler separately. OpenAI, Google, Microsoft and Anthropic have...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e37ced337db759f4743dfc3537eacfc6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e37ced337db759f4743dfc3537eacfc6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/openai-google-microsoft-and-anthropic-will-honor-one-robots-txt-signal-for-ai-training-opt-outs-srjmtpgd1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,openai,anthropic","timeRequired":"PT2M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"OpenAI, Google, Microsoft and Anthropic will honor one robots.txt signal for AI training opt-outs"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/openai-google-microsoft-and-anthropic-will-honor-one-robots-txt-signal-for-ai-training-opt-outs-srjmtpgd1#faq","mainEntity":[{"@type":"Question","name":"How do I block my website's content from being used in AI training with robots.txt?","acceptedAnswer":{"@type":"Answer","text":"Add a group to robots.txt using the new policy token `__usage_ai-training` and list the paths you want excluded. OpenAI, Google, Microsoft and Anthropic have said they will honor this single shared signal instead of requiring separate rules per crawler. The draft was written by Anthropic's Kevin Kelley and submitted to the IETF as an individual submission, so it currently has no formal standards status. Site operators weighing AI crawler policies can follow standards coverage like this on daily.dev."}},{"@type":"Question","name":"When will Google and Microsoft start honoring the AI training opt-out signal in robots.txt?","acceptedAnswer":{"@type":"Answer","text":"Microsoft starts on February 1, 2027, where the new signal replaces its existing NoArchive and NoCache tags. Google is targeting early 2027. OpenAI plans to support the signal in the coming months and will publish implementation details later, while Anthropic has already said it will honor it. Teams planning crawler and SEO configs around this rollout can track the timeline on daily.dev."}},{"@type":"Question","name":"Is the robots.txt AI training opt-out signal legally binding on crawlers?","acceptedAnswer":{"@type":"Answer","text":"No, the signal states only a preference, not a crawling order. Private agreements between a site and an AI company can override it. The proposal also has no formal standards status since it was submitted to the IETF as an individual submission rather than through a working group process. Developers deciding how much to trust crawler opt-outs can weigh the limits via daily.dev."}}]} ```