--- title: How a looped transformer architecture could let OpenAI's... description: OpenAI's rumored Astra model reportedly uses a 'recurrent depth' or looped transformer architecture, running data through the same layers multiple times per... canonical: https://daily.dev/posts/how-a-looped-transformer-architecture-could-let-openai-s-astra-punch-above-its-parameter-count-zkmicrmf8 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How a looped transformer architecture could let OpenAI's Astra punch above its parameter count | daily.dev og:description: OpenAI's rumored Astra model reportedly uses a 'recurrent depth' or looped transformer architecture, running data through the same layers multiple times per... og:url: https://daily.dev/posts/how-a-looped-transformer-architecture-could-let-openai-s-astra-punch-above-its-parameter-count-zkmicrmf8 og:image: https://api.daily.dev/og/posts/ZKMICrMF8.png og:image:alt: How a looped transformer architecture could let OpenAI's Astra punch above its parameter count og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How a looped transformer architecture could let OpenAI's Astra punch above its parameter count **[Collections](https://daily.dev/sources/collections)** · 4 min read · 2 upvotes · 0 comments ## Summary OpenAI's rumored Astra model reportedly uses a 'recurrent depth' or looped transformer architecture, running data through the same layers multiple times per token instead of a single fixed pass. This lets a smaller parameter model behave like a larger one, with research suggesting a 1.3–1.7x effective-parameter multiplier. The tradeoff is higher inference FLOPs and prefill latency in exchange for a smaller footprint, and reasoning happening in internal numerical states rather than visible chain-of-thought text makes model behavior harder to monitor from outside. ## Content ## What the reports actually say According to The Information, OpenAI's upcoming Astra model uses something called "recurrent depth" or a "looped transformer." The basic idea: instead of passing text through a fixed stack of transformer layers once, the model runs the same layers multiple times before producing the next token. More computation per token, without proportionally more parameters. The efficiency case is straightforward. One recurrent-depth paper found that running the same block twice gives roughly a 1.38x effective-parameter multiplier. So a 10T recurrent model might perform like a 13-17T standard model, while using less memory and bandwidth. That's a real advantage. OpenAI is reportedly putting limits on how many loops Astra runs. Whether those limits hold as the company chases more performance is a separate question. ## This isn't a novel architecture The looped transformer idea has been around for a while. Nanbeige 4.2-3B, an open-weight model released about two months ago, reuses a 22-layer stack twice to reach 44 effective layers without duplicating weights. The researchers found two passes gave the best trade-off, retaining about 75% of the token efficiency of a standard architecture. More passes gave barely any gains but made training much slower. The concept traces back to a NeurIPS paper on "Mixture-of-Recursions," which adds a learned router so easy tokens exit after one pass while harder tokens get more computation. Anthropic and Google DeepMind are reportedly already exploring similar approaches. So Astra may be a good model, but the looped transformer part is a modest architectural tweak, not a breakthrough. ## The safety concern is more interesting than the architecture Here's where things get genuinely complicated. The worry isn't really about looped transformers specifically. It's about what happens when more of a model's reasoning moves into internal numerical states rather than readable chain-of-thought text. Visible chain-of-thought has practical safety value. After the Hugging Face attack in July 2025, researchers were able to retrace what happened partly because the model's reasoning was legible. If reasoning moves into latent space, that visibility disappears. Ryan Greenblatt from Redwood Research flagged concern that Astra may solve hard problems "entirely in its head." OpenAI chief scientist Jakub Pachocki pushed back, warning against what he called confused reporting fueling a "race into unmonitorability." The awkward part: both of them co-authored a July 2025 position paper with roughly forty other researchers from OpenAI, DeepMind, Anthropic, Meta, Amazon, the UK AI Security Institute, and Redwood Research. That paper called chain-of-thought monitorability a fragile safety opportunity and urged standardized evaluations and reporting. They agree on the problem. They're disagreeing about whether Astra actually represents it. OpenAI has reportedly acknowledged that Astra shows a "substantial decrease in chain-of-thought monitorability compared to previous models," which is a notable admission regardless of the architectural explanation. ## What looped transformers actually do to reasoning visibility It's worth being precise here, because some of the reporting has been loose. Reusing layers does not by itself suppress visible chain-of-thought. It adds computation in hidden states before the next token is emitted, the same way ordinary transformer layers do. A deeper standard model would have the same effect. The more plausible concern is indirect: if a model uses more recurrent passes, it may need to generate fewer intermediate reasoning tokens. More computation happens in latent activations that can't be read as text. But this is a matter of degree, not a categorical shift caused by the looped architecture itself. There's also an open question about what monitoring even looks like as models become more multimodal. Chain-of-thought monitoring works when the predicted quantity is text. For robotics or image generation, there may not be a natural equivalent. ## Where this leaves things The EU's GPAI Code of Practice now requires signatories, including OpenAI, to submit model reports covering evaluations and sample inputs and outputs. That requirement matters more, not less, as visible reasoning text becomes scarcer. The looped transformer architecture is real and has genuine efficiency benefits. The safety concern about reasoning moving into latent space is also real, and predates Astra. Whether Astra specifically crosses a meaningful line, or whether the reporting overstated the architectural novelty, is still being sorted out by people who co-wrote the warning about exactly this problem. ## Questions this post answers ### What is a recurrent depth or looped transformer architecture? It is a transformer design where data passes through the same layers multiple times before producing the next token, instead of moving through a fixed stack of layers once. This increases the amount of computation applied per token without increasing the number of parameters, allowing a smaller model to behave more like a larger one at the cost of higher inference compute. _Curious readers tracking emerging LLM architectures can follow deep dives like this on daily.dev._ ### How much of a performance boost does looping a transformer block twice provide? Research on recurrent depth models found that running the same recurrent block twice gives roughly a 1.38x effective-parameter multiplier, with a plausible range of 1.3 to 1.7x depending on implementation. For example, a 10 trillion parameter model could perform like a 13.8 trillion parameter one under this approach. _Anyone evaluating model scaling tradeoffs can dig into architecture analysis like this on daily.dev._ ### What are the downsides of using recurrent depth architecture in a large language model? Inference FLOPs and prefill latency increase because the same layers are run multiple times per token, even though pre-training compute and decode latency stay roughly the same under implementations like recirculation. It also makes model reasoning less transparent, since more computation happens in internal numerical states rather than explicit, human-readable chain-of-thought text. _Developers weighing model transparency against efficiency gains can find analysis like this on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 3 comments across x (as of 2026-09-06). **TL;DR:** The limited discussion available focuses less on Astra itself and more on the broader challenge of monitoring reasoning once it moves beyond text, with commenters skeptical that current chain-of-thought monitoring approaches generalize. **Sentiment:** 5% positive · 15% mixed · 80% skeptical **The pushback** - Monitoring approaches built for text-based reasoning traces may not work once actions involve video frames, robot torque, or sensor data. - Systems could end up fabricating a plausible explanation after the fact rather than producing a genuine, auditable trace. **By community** - x (skeptical): Replies question whether any monitoring scheme can meaningfully audit non-text reasoning traces like robot trajectories or sensor data. **Open questions** - What would a valid 'trace' even look like for multimodal or embodied actions rather than text-based reasoning? **Highlights** > @ericjang11 The hard part is what counts as a 'trace' once the action is a video frame or a robot torque. Text CoT monitors; multimodal runs need sensor-level audits or they just invent a story after the fact. > — [yandt888 on x](https://x.com/yandt888/status/2096658859440169443) > @ericjang11 word space monitors fall over on robot traces. you end up watching trajectories, contact, and energy, not sentences > — [shzhv13 on x](https://x.com/shzhv13/status/2096683645809369581) **Source threads** - [x](https://x.com/ericjang11/status/2096647141267538126) · 0 points · 3 comments --- Tags: [#deep-learning](https://daily.dev/tags/deep-learning), [#openai](https://daily.dev/tags/openai), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/how-a-looped-transformer-architecture-could-let-openai-s-astra-punch-above-its-parameter-count-zkmicrmf8) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How a looped transformer architecture could let OpenAI's Astra punch above its parameter count","url":"https://daily.dev/posts/how-a-looped-transformer-architecture-could-let-openai-s-astra-punch-above-its-parameter-count-zkmicrmf8","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-a-looped-transformer-architecture-could-let-openai-s-astra-punch-above-its-parameter-count-zkmicrmf8"},"datePublished":"2026-09-02T02:29:26.292Z","dateModified":"2026-09-06T20:11:21.097Z","description":"OpenAI's rumored Astra model reportedly uses a 'recurrent depth' or looped transformer architecture, running data through the same layers multiple times per...","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-a-looped-transformer-architecture-could-let-openai-s-astra-punch-above-its-parameter-count-zkmicrmf8","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"deep-learning,openai,ai-inference","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"How a looped transformer architecture could let OpenAI's Astra punch above its parameter count"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-a-looped-transformer-architecture-could-let-openai-s-astra-punch-above-its-parameter-count-zkmicrmf8#faq","mainEntity":[{"@type":"Question","name":"What is a recurrent depth or looped transformer architecture?","acceptedAnswer":{"@type":"Answer","text":"It is a transformer design where data passes through the same layers multiple times before producing the next token, instead of moving through a fixed stack of layers once. This increases the amount of computation applied per token without increasing the number of parameters, allowing a smaller model to behave more like a larger one at the cost of higher inference compute. Curious readers tracking emerging LLM architectures can follow deep dives like this on daily.dev."}},{"@type":"Question","name":"How much of a performance boost does looping a transformer block twice provide?","acceptedAnswer":{"@type":"Answer","text":"Research on recurrent depth models found that running the same recurrent block twice gives roughly a 1.38x effective-parameter multiplier, with a plausible range of 1.3 to 1.7x depending on implementation. For example, a 10 trillion parameter model could perform like a 13.8 trillion parameter one under this approach. Anyone evaluating model scaling tradeoffs can dig into architecture analysis like this on daily.dev."}},{"@type":"Question","name":"What are the downsides of using recurrent depth architecture in a large language model?","acceptedAnswer":{"@type":"Answer","text":"Inference FLOPs and prefill latency increase because the same layers are run multiple times per token, even though pre-training compute and decode latency stay roughly the same under implementations like recirculation. It also makes model reasoning less transparent, since more computation happens in internal numerical states rather than explicit, human-readable chain-of-thought text. Developers weighing model transparency against efficiency gains can find analysis like this on daily.dev."}}]} ```