A detailed case study walks through building a production RAG platform for a New York VC fund that unifies PDFs, emails, transcripts, and news into a searchable knowledge base with LLM agents for meeting briefings, research, and Q&A. It covers the storage architecture (S3, Pinecone, OpenSearch, Neo4j, Redis, PostgreSQL), a seven-stage ingestion pipeline, a mid-project rebuild of the agent layer from a class hierarchy into a configurable PromptLayerAgent, Airflow automation DAGs, and a production incident where Airflow silently failed for 20 days due to an upstream bug, prompting monitoring and reliability improvements. It closes with honest limitations and a roadmap toward MCP-style tool exposure and agent skills.
Table of contents
Where it started, and what changedThe storage layerTurning documents into a knowledge baseThe agent layer, and rebuilding it mid-projectAutomation: the Airflow DAGsPrompt experimentation and PromptLayerTwo faces: WebApp and Frontend DemoWhat running it in production taught usHonest limitationsWhere it goes nextQuestions this post answers
How can retrieval accuracy on table-heavy documents in a RAG pipeline be improved?
Serializing table rows into self-contained text blocks with headers, units, currencies, titles, footnotes, and surrounding context, using a model like GPT-4o-mini, substantially improves retrieval. In one production system, this pushed table-question retrieval accuracy from roughly 70% to about 94%, and chart-related question accuracy to about 95%, versus naive chunking that splits tables and loses their headers. Explore more real-world RAG architecture patterns like this on daily.dev before designing your own table-heavy pipeline.
Why would a team replace a class-hierarchy based AI agent system with a single template-driven agent?
A class hierarchy of specialized agents required a code change and redeploy for every new behavior or tweak, which locked non-engineers out of improving agent reasoning. Replacing it with one universal agent driven by configurable prompt templates (system prompt, user prompt, context calls, and callable functions) let non-technical staff edit behavior directly through a prompt management UI without redeploying code. Compare agent architecture trade-offs like this one on daily.dev when deciding how to structure your own LLM agent layer.
What causes an Airflow DAG to appear healthy while actually being stuck for weeks?
A worker restart triggered by an out-of-memory condition can leave a DAG stuck without the scheduler ever marking the task as failed, so a basic health endpoint reports everything as fine. This matched a known upstream Airflow bug (issue #55029) and went unnoticed for over 20 days until StatsD-to-CloudWatch metrics, DAG run timeouts, and SNS-backed failure alerts were added after upgrading to Airflow 3.1.0. Track production reliability issues like this Airflow bug on daily.dev before they cost you weeks of silent downtime.