A case study demonstrates an architecture where a Phoenix LiveView support agent keeps raw customer identity (emails, phone numbers, names) inside the application, while the Jido agent framework and OpenAI's model only ever see stable pseudonyms generated by a library called Obscura. The write-up walks through every place personal data could leak in an agent workflow beyond the initial prompt: tool arguments, tool results, streamed responses, credentials, and logs/telemetry. A trusted tool restores a raw email only briefly for a local lookup, then re-protects the result before returning it to the model. Streaming responses stay tokenized until the final answer is restored client-side. OpenAI API keys are stored as opaque references and resolved only at the Finch HTTP layer. The demo includes tests verifying no raw values leak through any of these boundaries, while explicitly noting limits: it doesn't provide authorization, DLP, or regulatory compliance, and can't stop a model from generating new sensitive values outside the vault.
Table of contents
The Result in One MinuteAn Agent Has More Than One ExitThe Ownership RuleCreate One Vault for the SessionStable Pseudonyms Preserve UtilityA Trusted Tool Restores One IdentifierTool Results Are Another Outbound BoundaryStreaming Must Not Restore Too EarlyCredentials Are a Separate BoundaryLogs and Telemetry Need Their Own PolicyWhat the Evidence ProvesWhat the Evidence Does Not ProveWhere This Pattern Is UsefulRun the Complete DemonstrationThe Model Does Not Need the NameQuestions this post answers
How do you prevent an LLM agent from receiving raw customer PII like email addresses when tool calls are involved?
Pseudonymize identifiers before they reach the model, and have trusted tools restore the raw value only briefly for a local lookup before re-protecting the result. In an Elixir example using Jido, ReqLLM, and Obscura, an email is tokenized as <<EMAIL_001>> at the prompt boundary, restored inside a tool's local lookup, then replaced with a token again before the tool result returns to the agent, so the model never sees the actual address. Teams designing agent tool pipelines that touch PII can track patterns like this via daily.dev.
How can I keep streaming LLM responses from leaking sensitive data before the full answer is ready?
Keep the streamed content tokenized throughout the stream and only restore known values once the complete provider answer is returned. In this architecture, progress events carry only protected text and provider deltas, and restoration happens after the full response arrives, producing a tokenized provider answer and a separately restored trusted UI response. Developers building streaming AI agent UIs can follow implementation details like this on daily.dev.
What is the safest way to pass an OpenAI API key through an agent framework like Jido without exposing it in state?
Store the key in a private ETS table on a single process and pass only an opaque reference plus a placeholder through the agent and model layers. A ReqLLM Finch adapter resolves the real key immediately before transmission, checks the destination against an approved host and path, and rejects the request before it reaches the network if the reference has expired or the destination isn't approved. Engineers securing credentials inside LLM agent pipelines can find architecture writeups like this on daily.dev.