Two research papers examine contextual integrity in LLMs — the ability to control what personal information is appropriately shared in different task contexts. CIMemories, a new benchmark using synthetic user profiles with 100+ attributes, finds frontier models leak sensitive information inappropriately up to 69% of the time, with GPT-5's violation rate climbing from 0.1% to 25.1% as tasks and repeated runs accumulate. Privacy-conscious prompting fails to fix this, as models tend to overgeneralize into sharing everything or nothing. A second paper proposes a reinforcement learning framework that explicitly trains models to reason about context before disclosing information, showing reduced leakage while preserving task performance, with gains transferring to the established PrivacyLens benchmark.
Questions this post answers
How often do LLMs like GPT-5 leak private information from memory when performing unrelated tasks?
GPT-5's rate of inappropriate information disclosure rises from 0.1% after a single task to 9.6% after 40 tasks, and reaches 25.1% when the same prompt is run 5 times, according to the CIMemories benchmark. This benchmark tests synthetic user profiles with over 100 attributes each against diverse task contexts, finding frontier models exhibit up to 69% attribute-level violations overall. Track how memory features in AI coding assistants affect data leakage risk with daily.dev.
Does telling an LLM to be privacy-conscious in the prompt prevent it from leaking sensitive data?
No, privacy-conscious prompting does not solve inappropriate information disclosure. Models tend to overgeneralize, either sharing everything or withholding everything, rather than making nuanced, context-dependent decisions about what information is appropriate to reveal for a specific task. Developers weighing privacy trade-offs in LLM-based tools can follow this research on daily.dev.
Can reinforcement learning reduce inappropriate information disclosure by AI agents?
Yes, a reinforcement learning framework trained on just 700 synthetic examples with diverse contexts and disclosure norms substantially reduced inappropriate information disclosure while maintaining task performance across multiple model sizes and families. The improvements transferred to established contextual integrity benchmarks like PrivacyLens, which uses human annotations to evaluate privacy leakage in AI assistant actions and tool calls. Anyone building safer autonomous agents can follow emerging privacy-preserving techniques on daily.dev.
319 Impressions1 Comment