--- title: robots.txt in the age of AI crawlers | daily.dev description: A practical guide to robots.txt covering both the classic syntax rules and the new wave of AI crawlers. Explains the three key precedence rules (one group per... canonical: https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: robots.txt in the age of AI crawlers | daily.dev og:description: A practical guide to robots.txt covering both the classic syntax rules and the new wave of AI crawlers. Explains the three key precedence rules (one group per... og:url: https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee og:image: https://api.daily.dev/og/posts/NNwKBP4Ee.png og:image:alt: robots.txt in the age of AI crawlers og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # robots.txt in the age of AI crawlers **[Flavio Copes](https://daily.dev/sources/flaviocopes)** · 6 min read · 1 upvotes · 1 comments ## Summary A practical guide to robots.txt covering both the classic syntax rules and the new wave of AI crawlers. Explains the three key precedence rules (one group per crawler, longest rule wins, no match means allowed), lists major AI bots including GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, and Google-Extended, and draws a critical distinction between training crawlers and AI search crawlers. Provides a ready-to-use robots.txt template that blocks training crawlers while keeping AI search indexing open. Also covers compliance realities — robots.txt is a convention, not enforcement — and the strategic tradeoff of blocking AI crawlers when AI search is increasingly where readers discover content. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I block GPTBot from training on my site without blocking regular search engines? Add a separate User-agent: GPTBot group with Disallow: / in robots.txt, then keep a User-agent: * group with Allow: / for everyone else. Because a crawler matches only its most specific User-agent group and groups never combine, GPTBot follows its own block while other crawlers fall through to the general rule. _Developers tuning crawler access for AI training keep configuration references like this handy on daily.dev._ ### Does OpenAI's ChatGPT-User agent respect robots.txt Disallow rules? Not reliably. OpenAI explicitly states that robots.txt rules may not apply to ChatGPT-User, the agent that fetches a page in response to a live user request, unlike GPTBot (training) or OAI-SearchBot (search indexing), which do follow published robots.txt guidance. _Anyone deciding how to handle AI crawlers can track these distinctions on daily.dev._ ### What does blocking Google-Extended in robots.txt actually do? Blocking Google-Extended tells Google not to use a site's content for training Gemini models, without affecting Google Search ranking. It is not a separate crawler but a control flag, distinct from Googlebot, allowing site owners to opt out of AI training data collection while remaining fully indexed for search. _Site owners weighing AI training opt-outs against search visibility can follow this nuance on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@rizzdev** · 0 upvotes > Blocking training bots while leaving AI search paths open is the only robots.txt split that still prices discovery against scrape risk ## Similar posts on daily.dev - [Every AI Crawler in 2026: The Reference Table \(And Which Ones Your WAF Is Silently Blocking\)](https://daily.dev/posts/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking--ujfhbfivn) · Security Boulevard · 1 upvotes · 1 comments - [Fix Your robots.txt or Your Site Disappears from Google](https://daily.dev/posts/fix-your-robots-txt-or-your-site-disappears-from-google-iidz6yzyo) · Hacker News · 0 upvotes · 0 comments - [Your website's next visitor is an AI agent](https://daily.dev/posts/your-website-s-next-visitor-is-an-ai-agent-pc83koj9u) · RUBYLAND · 1 upvotes · 0 comments - [Publishers say no to AI scrapers, block bots at server level](https://daily.dev/posts/publishers-say-no-to-ai-scrapers-block-bots-at-server-level-nite7hp90) · The Register · 1 upvotes · 0 comments - [Your site, your rules: new AI traffic options for all customers](https://daily.dev/posts/your-site-your-rules-new-ai-traffic-options-for-all-customers-zrn0fx6z7) · Cloudflare · 0 upvotes · 0 comments --- Tags: [#webdev](https://daily.dev/tags/webdev), [#machine-learning](https://daily.dev/tags/machine-learning), [#seo](https://daily.dev/tags/seo) [View this post on daily.dev](https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"robots.txt in the age of AI crawlers","url":"https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee"},"datePublished":"2026-08-07T06:25:51.909Z","dateModified":"2026-09-14T07:17:38.360Z","description":"A practical guide to robots.txt covering both the classic syntax rules and the new wave of AI crawlers. Explains the three key precedence rules (one group per...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7219243bb6b6113ecc90bd933965a4ef?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7219243bb6b6113ecc90bd933965a4ef?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Flavio Copes","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Flavio Copes","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/1f2dbd275797411bbd37bfdaaf0951a9","url":"https://daily.dev/sources/flaviocopes"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"webdev,machine-learning,seo","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Flavio Copes","item":"https://daily.dev/sources/flaviocopes"},{"@type":"ListItem","position":3,"name":"robots.txt in the age of AI crawlers"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee","comment":[{"@type":"Comment","text":"Blocking training bots while leaving AI search paths open is the only robots.txt split that still prices discovery against scrape risk","datePublished":"2026-08-09T00:39:21.950Z","url":"https://daily.dev/posts/NNwKBP4Ee#c-jeMhlbaQE","author":{"@type":"Person","name":"Andrew","url":"https://daily.dev/rizzdev","image":"https://media.daily.dev/image/upload/s--35vRfXAA--/f_auto/v1785413727/avatars/avatar_fQ7ttCKtPuu6tDvcgQXv5?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/robots-txt-in-the-age-of-ai-crawlers-nnwkbp4ee#faq","mainEntity":[{"@type":"Question","name":"How do I block GPTBot from training on my site without blocking regular search engines?","acceptedAnswer":{"@type":"Answer","text":"Add a separate User-agent: GPTBot group with Disallow: / in robots.txt, then keep a User-agent: * group with Allow: / for everyone else. Because a crawler matches only its most specific User-agent group and groups never combine, GPTBot follows its own block while other crawlers fall through to the general rule. Developers tuning crawler access for AI training keep configuration references like this handy on daily.dev."}},{"@type":"Question","name":"Does OpenAI's ChatGPT-User agent respect robots.txt Disallow rules?","acceptedAnswer":{"@type":"Answer","text":"Not reliably. OpenAI explicitly states that robots.txt rules may not apply to ChatGPT-User, the agent that fetches a page in response to a live user request, unlike GPTBot (training) or OAI-SearchBot (search indexing), which do follow published robots.txt guidance. Anyone deciding how to handle AI crawlers can track these distinctions on daily.dev."}},{"@type":"Question","name":"What does blocking Google-Extended in robots.txt actually do?","acceptedAnswer":{"@type":"Answer","text":"Blocking Google-Extended tells Google not to use a site's content for training Gemini models, without affecting Google Search ranking. It is not a separate crawler but a control flag, distinct from Googlebot, allowing site owners to opt out of AI training data collection while remaining fully indexed for search. Site owners weighing AI training opt-outs against search visibility can follow this nuance on daily.dev."}}]} ```