Greg Brockman

IBM Technology

jason liu

<p>OpenAI has developed GPT-Red, an automated red-teaming model trained to attack its own AI systems and find prompt injection vulnerabilities before deployment. It’s not being released publicly.</p>
<h2>How it works</h2>
<p>GPT-Red was trained using self-play reinforcement learning: an attacker model probes AI systems while defender models push back. Over roughly a year of development, this loop produced a model that’s significantly better at finding vulnerabilities than human red-teamers — succeeding in 84% of indirect prompt injection scenarios compared to 13% for humans.</p>
<p>The training covered a range of real-world scenarios: email, webpages, files, and tool outputs. In practice, GPT-Red generates thousands of exploit variations to map where a model’s defenses break down.</p>
<h2>The “fake chain of thought” attack</h2>
<p>GPT-Red’s most notable discovery is a new attack class called “fake chain of thought.” It works by planting false information in a model’s working memory — essentially spoofing the model’s own reasoning process so it acts on fabricated premises. Against older GPT-5 models, over 90% of GPT-Red’s strongest attacks succeeded. That’s a bad number.</p>
<p>Real-world tests were more concrete than typical benchmark results. GPT-Red convinced a production vending machine management agent to slash prices and exfiltrate data from a Codex CLI agent. These aren’t theoretical edge cases.</p>
<h2>What it did to GPT-5.6</h2>
<p>OpenAI used GPT-Red’s adversarial data to harden GPT-5.6. The results are significant: fewer than 23% of GPT-Red’s attacks succeeded against GPT-5.6, down from over 90% against GPT-5. GPT-5.6 Sol specifically achieved a 0.05% attack success rate against GPT-Red’s direct attacks — roughly six times fewer failures than the strongest production model released four months earlier.</p>
<p>According to OpenAI, the security improvements didn’t increase over-refusals, which is the usual tradeoff when you try to make models more cautious.</p>
<p>GPT-Red’s adversarial data has fed into every OpenAI model since GPT-5.3.</p>
<h2>Limitations</h2>
<p>GPT-Red isn’t a complete solution. It’s still weak against multi-turn attacks and instructions embedded in images. OpenAI says it plans to scale the system further and expects to publish a technical preprint.</p>
<h2>Why it’s not being released</h2>
<p>OpenAI cited the offensive capabilities, the significant compute required, and the roughly year-long development investment as reasons for keeping GPT-Red internal. Given that it cracks older models at a 90%+ rate, that reasoning is hard to argue with.</p>


Collections

OpenAI has built GPT-Red, an internal LLM designed to attack its own models and discover prompt injection vulnerabilities. It operates via a self-play loop where it probes for weaknesses while other models defend. GPT-Red discovered a notable 'fake chain of thought' attack that tricks models into acting on spoofed reasoning. Against GPT-5, over 90% of its strongest attacks succeeded, but adversarial training using GPT-Red's findings reduced that rate to below 23% on GPT-5.6. OpenAI won't release GPT-Red publicly due to high compute costs and roughly a year of development time.

OpenAI built an internal red-teaming AI called GPT-Red to harden its models against prompt injection