<h1>Architecture Deep-Dive: Building a Multi-Agent AI Workspace with Persistent Memory</h1>
How I built a self-hosted alternative to ChatGPT Teams + Cursor in 3 months, and the architectural decisions that made it possible.
<hr>
<h2>The Problem</h2>
After helping multiple companies evaluate AI tools, I noticed a pattern: teams want AI agents, but their compliance team blocks SaaS solutions. The available alternatives were either:
<ul>
<li>Cloud-only tools (Cursor, ChatGPT Enterprise) — data leaves your VPC</li>
<li>Basic chat wrappers (Open WebUI, AnythingLLM) — no agents, no memory, no workflow</li>
<li>Enterprise solutions ($50k+/year) — Glean, Hebbia, custom enterprise platforms</li>
</ul>
There was a gap: a production-ready, self-hosted, multi-agent AI workspace that didn’t require a quarter-million-dollar contract.
So I built <a href="https://github.com/aliyevaladddin/AladdinAI" target="_blank" rel="noopener nofollow">AladdinAI</a>.
In this post, I’ll walk through the architectural decisions I made — what worked, what didn’t, and what I’m still uncertain about.
<hr>
<h2>Architecture Overview</h2>
<pre><code>┌─────────────────────────────────────────────────┐
│ Frontend (Next.js 15 + React 19) │
│ - Agent UI, CRM, settings │
│ - SSE for streaming responses │
└──────────────┬──────────────────────────────────┘
 │ REST + SSE
┌──────────────▼──────────────────────────────────┐
│ Backend (FastAPI + SQLAlchemy 2 async) │
│ │
│ ┌─────────────────────────────────────────┐ │
│ │ Agent Runner (orchestrator) │ │
│ │ ├─ LLM Service (provider-agnostic) │ │
│ │ ├─ Tools Registry (@tool decorator) │ │
│ │ ├─ Memory Service (Mongo + vectors) │ │
│ │ ├─ Safety Stack (NemoGuard, GLiNER) │ │
│ │ └─ Gates (pre/post execution) │ │
│ └─────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────┐ │
│ │ Channels │ │
│ │ Telegram | WhatsApp | Email | SMS │ │
│ └─────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────┐ │
│ │ Terminal Providers (Docker + Traefik) │ │
│ │ ttyd (local) | wetty (SSH) │ │
│ └─────────────────────────────────────────┘ │
└──────────────┬──────────────────────────────────┘
 │
 ┌───────┴────────┐
 │ │
┌──────▼──────┐ ┌──────▼─────────────┐
│ Postgres │ │ MongoDB Atlas │
│ (state) │ │ (vectors+memory) │
└─────────────┘ └────────────────────┘
</code></pre>
The key design principle: separation between canonical state (Postgres) and semantic recall (MongoDB).
<hr>
<h2>Memory Architecture</h2>
The memory layer is the part I’m most proud of — and most uncertain about.
<h3>Three-Tier Memory Model</h3>
Every user has three types of memory storage:
<pre><code># From services/memory.py

PRIVATE_COLLECTION = &quot;agent_memories&quot;
SHARED_COLLECTION = &quot;shared_context&quot;
SUMMARY_COLLECTION = &quot;conversation_summaries&quot;

EMBED_MODEL = &quot;nvidia/llama-3.2-nv-embedqa-1b-v2&quot;
EMBED_DIM = 2048
</code></pre>
<ol>
<li>Private memory (per-agent): Facts only this specific agent should know</li>
<li>Shared context (per-user): Facts all agents under this user can access</li>
<li>Conversation summaries: Rolled-up chat history (no vectors yet)</li>
</ol>
<h3>Why This Layout?</h3>
The split solves a real problem: if Agent A learns “the user is allergic to peanuts” during a personal conversation, should Agent B (the work assistant) know that?
The answer depends on scope. Some facts are universal (<code>user.name = &quot;Alex&quot;</code>), some are agent-specific (<code>prefers formal tone in this conversation</code>).
By making the agent explicitly decide via tool calls, we get:
<ul>
<li>Clear data ownership</li>
<li>Easy GDPR compliance (delete by agent_id or user_id)</li>
<li>No information leaks between agent personas</li>
</ul>
<h3>Vector Search Setup</h3>
I went with MongoDB Atlas Vector Search instead of Pinecone/Weaviate. The reasoning:
✅ One database: structured CRM data + vector embeddings live together
✅ No sync layer: avoiding the “two sources of truth” problem
✅ Atlas Vector Search is good enough: cosine similarity, filtering, sub-100ms recall
❌ Vendor lock-in: real concern, considering pgvector as self-hosted alternative
<h3>Memory Extraction Pipeline</h3>
After each user message, I run an extraction pipeline:
<pre><code># From services/extraction.py

&quot;&quot;&quot;Per-message memory extraction.

After an agent finishes a turn, optionally distill the exchange into a small
set of facts and persist them via the same gates/safety pipeline that backs
the explicit `remember` tool.

Config lives in `agent.tools_config.extraction`:

 {
 &quot;extraction&quot;: {
 &quot;enabled&quot;: true,
 &quot;model&quot;: &quot;&lt;nim model id&gt;&quot; | null,
 &quot;max_facts&quot;: 5
 }
 }
&quot;&quot;&quot;
</code></pre>
The flow:
<ol>
<li>User sends message → Agent responds</li>
<li>Extraction LLM call analyzes the exchange</li>
<li>Returns 0-5 facts as structured JSON</li>
<li>Each fact goes through safety stack (PII redaction, content moderation)</li>
<li>Approved facts get embedded and stored</li>
</ol>
Trade-off I’m wrestling with: This runs per message, which is expensive at scale. Considering:
<ul>
<li>Batch extraction every N messages</li>
<li>Smaller local model for extraction (Llama 3 8B instead of full)</li>
<li>Skip extraction for trivial messages</li>
</ul>
If you’ve solved this differently, I’d love to hear how.
<hr>
<h2>Plugin System</h2>
I went all-in on decorators. Three plugin types:
<h3>1. Tools (<code>@tool</code> decorator)</h3>
<pre><code># From tools/base.py

@dataclass
class ToolContext:
 db: AsyncSession
 user_id: int
 agent_id: int | None = None
 session_id: int | None = None
 extra: dict[str, Any] = field(default_factory=dict)


@dataclass
class Tool:
 name: str
 description: str
 parameters: dict[str, Any]
 func: ToolFunc

 def openai_schema(self) -&gt; dict[str, Any]:
 return {
 &quot;type&quot;: &quot;function&quot;,
 &quot;function&quot;: {
 &quot;name&quot;: self.name,
 &quot;description&quot;: self.description,
 &quot;parameters&quot;: self.parameters,
 },
 }
</code></pre>
Adding a new tool:
<pre><code>@tool(
 name=&quot;search_contacts&quot;,
 description=&quot;Search CRM contacts by name or email&quot;,
 parameters={
 &quot;type&quot;: &quot;object&quot;,
 &quot;properties&quot;: {
 &quot;query&quot;: {&quot;type&quot;: &quot;string&quot;}
 },
 &quot;required&quot;: [&quot;query&quot;]
 }
)
async def search_contacts(ctx: ToolContext, query: str) -&gt; list[dict]:
 # ctx.user_id is automatically scoped
 # ctx.db is ready to use
 ...
</code></pre>
The decorator handles:
<ul>
<li>Auto-discovery on import</li>
<li>JSON Schema generation for OpenAI function calling</li>
<li>User-scoped database queries (no cross-tenant leaks)</li>
</ul>
<h3>2. Terminal Adapters (<code>@terminal_adapter</code> decorator)</h3>
For browser-based terminals (ttyd, wetty):
<pre><code>@terminal_adapter(&quot;ttyd&quot;)
class TtydAdapter(TerminalAdapter):
 def build_container_spec(self, *, provider_id, user_id, image, manifest, config):
 # Returns Docker container spec
 ...
 
 def build_session_url(self, *, provider_id, url_template, scheme, host, token):
 # Returns user-facing iframe URL
 ...
</code></pre>
Same pattern as tools — one decorator, auto-registered, type-safe.
<h3>3. Gates (pre/post execution)</h3>
Gates intercept agent execution at key points:
<pre><code>@gate(&quot;memory.read.before&quot;)
async def log_memory_reads(ctx: GateContext, query: str) -&gt; GateDecision:
 # Decide whether to allow this memory read
 # Log the decision
 return GateDecision.ALLOW
</code></pre>
This is how the safety stack hooks into agent execution without modifying the runner.
<hr>
<h2>Safety Stack</h2>
Three layers of defense:
<pre><code># Ingress (user → agent)
- NemoGuard: prompt injection detection
- GLiNER: PII detection and redaction
- Llama-Guard: harmful content filtering

# Egress (agent → user)
- Same stack, applied to agent responses
- Custom rules per agent (configurable)

# Tool execution
- Gates evaluate every tool call
- Audit log to gate_log.py
</code></pre>
The clever bit: PII redaction phases. Different operations have different PII tolerance:
<pre><code># Config-driven phases
safety_pii(phase=&quot;ingress&quot;) # Detect &amp; flag, don&#x27;t block
safety_pii(phase=&quot;egress&quot;) # Block if detected
safety_pii(phase=&quot;memory_write&quot;) # Redact before storing
safety_pii(phase=&quot;memory_read&quot;) # Pass through (already filtered)
</code></pre>
This avoids the common mistake of either being too paranoid (blocking everything) or too permissive (leaking secrets).
<hr>
<h2>Terminal Providers</h2>
The terminal subsystem went through several iterations. Final architecture:
<pre><code>Browser (iframe)
 │
 │ HTTP/WebSocket
 │
Traefik (forward-auth + path routing)
 │
 │ Internal docker network
 │
Container (ttyd OR wetty)
 │
 │ PTY OR SSH
 │
Target shell
</code></pre>
Forward-auth pattern:
<ol>
<li>Backend issues short-lived token (5 min, HMAC-signed)</li>
<li>URL includes <code>?token=xxx</code></li>
<li>Traefik calls <code>/api/terminal/auth</code> with the token</li>
<li>Backend validates, returns session cookie</li>
<li>Subsequent requests (CSS, JS, WebSocket upgrade) use the cookie</li>
</ol>
This avoids exposing terminal containers directly while keeping the user experience seamless.
The Pydantic + decorator + FSM stack might be overkill for 2 providers — but it makes adding new ones (Apache Guacamole, code-server) a 2-file change.
<hr>
<h2>LLM Abstraction</h2>
Provider-agnostic from day one:
<pre><code>LLMService
├── NVIDIA NIM (default, free, no rate limits)
├── OpenAI
├── Anthropic
├── Local models (via BentoML)
└── Per-agent fallback chains
</code></pre>
The killer feature: fallback chains. If your primary model is rate-limited, the agent automatically retries with the next in the chain. No errors visible to the user.
NVIDIA NIM is the default because:
<ul>
<li>Free with NVIDIA developer account</li>
<li>No rate limits (vs OpenAI’s hard caps)</li>
<li>Self-hostable via Docker</li>
<li>Good model selectio</li>
</ul>