--- title: Building a RAG Investment Assistant | daily.dev description: A detailed case study walks through building a production RAG platform for a New York VC fund that unifies PDFs, emails, transcripts, and news into a... canonical: https://daily.dev/posts/building-a-rag-investment-assistant-7trsiwger twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building a RAG Investment Assistant | daily.dev og:description: A detailed case study walks through building a production RAG platform for a New York VC fund that unifies PDFs, emails, transcripts, and news into a... og:url: https://daily.dev/posts/building-a-rag-investment-assistant-7trsiwger og:image: https://api.daily.dev/og/posts/7TrSiWGEr.png og:image:alt: Building a RAG Investment Assistant og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building a RAG Investment Assistant **[Serokell](https://daily.dev/sources/serokell)** · 16 min read · 0 upvotes · 0 comments ## Summary A detailed case study walks through building a production RAG platform for a New York VC fund that unifies PDFs, emails, transcripts, and news into a searchable knowledge base with LLM agents for meeting briefings, research, and Q&A. It covers the storage architecture (S3, Pinecone, OpenSearch, Neo4j, Redis, PostgreSQL), a seven-stage ingestion pipeline, a mid-project rebuild of the agent layer from a class hierarchy into a configurable PromptLayerAgent, Airflow automation DAGs, and a production incident where Airflow silently failed for 20 days due to an upstream bug, prompting monitoring and reliability improvements. It closes with honest limitations and a roadmap toward MCP-style tool exposure and agent skills. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can table data be chunked for RAG without losing headers and numeric context? Instead of naively splitting large tables across chunks (which loses headers and the relationships that make numbers meaningful), GPT-4o-mini can convert table rows into self-contained text blocks that carry their headers, units, currencies, title, footnotes, and surrounding context. This approach raised retrieval accuracy on table-related questions from about 70% to about 94%, and pushed chart-related question accuracy to about 95%. _daily.dev surfaces practical RAG engineering techniques like this for teams tuning retrieval accuracy._ ### Why would an Airflow deployment stay unhealthy for weeks without anyone noticing? A worker restart triggered by an out-of-memory condition can leave a DAG stuck without the scheduler ever marking the task as failed, and if the health endpoint doesn't check for that specific failure mode, the outage goes unreported. One deployment stayed broken for over 20 days due to a known upstream Airflow bug (issue #55029) before StatsD-to-CloudWatch metrics, DAG run timeouts, and SNS-backed alerts were added to catch it. _engineers hardening Airflow monitoring can find similar production incident writeups on daily.dev._ ### How can you combine vector search and keyword search for hybrid retrieval in a RAG system? Pair a vector database like Pinecone for semantic ('soft') queries with a search engine like OpenSearch enforcing 'strict' must-contain, exclude, and filter conditions, since keyword and metadata filtering catches what vector similarity misses. In one production query, 500 Pinecone hits were narrowed down to just 8 relevant documents after OpenSearch filtering was applied. _developers designing hybrid retrieval pipelines can compare real-world RAG architectures on daily.dev._ ## Similar posts on daily.dev - [Advanced AI for Finance and Tax: A Deep Dive into an Agentic RAG System](https://daily.dev/posts/advanced-ai-for-finance-and-tax-a-deep-dive-into-an-agentic-rag-system-tjaxjr8q1) · Coins Bench · 1 upvotes · 0 comments - [Building a Production RAG Assistant on DigitalOcean](https://daily.dev/posts/building-a-production-rag-assistant-on-digitalocean-nphu6g3iz) · DigitalOcean Community · 0 upvotes · 0 comments - [Building a RAG Tool in Ruby 4: What Actually Happened](https://daily.dev/posts/building-a-rag-tool-in-ruby-4-what-actually-happened-omgjhrdhw) · RUBYLAND · 0 upvotes · 0 comments - [Building a Practical AI Agent with RAG, MCP, and Ollama How I built \(and iterated on\) a production‑ready AI agent that actually works with real data](https://daily.dev/posts/building-a-practical-ai-agent-with-rag-mcp-and-ollama-how-i-built-and-iterated-on-a-production-r-1v9gmzmqj) · The Miners · 2 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#rag](https://daily.dev/tags/rag), [#vector-search](https://daily.dev/tags/vector-search), [#apache-airflow](https://daily.dev/tags/apache-airflow), [#pinecone](https://daily.dev/tags/pinecone) [View this post on daily.dev](https://daily.dev/posts/building-a-rag-investment-assistant-7trsiwger) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building a RAG Investment Assistant","url":"https://daily.dev/posts/building-a-rag-investment-assistant-7trsiwger","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-a-rag-investment-assistant-7trsiwger"},"datePublished":"2026-08-18T21:31:57.370Z","dateModified":"2026-09-14T07:06:07.506Z","description":"A detailed case study walks through building a production RAG platform for a New York VC fund that unifies PDFs, emails, transcripts, and news into a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5f18a3d7c186636bba42f0c3ceac604d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5f18a3d7c186636bba42f0c3ceac604d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Serokell","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Serokell","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/e9d268330ce14343a9e3875ff97aa4c3","url":"https://daily.dev/sources/serokell"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/building-a-rag-investment-assistant-7trsiwger","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,rag,vector-search,apache-airflow,pinecone","timeRequired":"PT16M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Serokell","item":"https://daily.dev/sources/serokell"},{"@type":"ListItem","position":3,"name":"Building a RAG Investment Assistant"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/building-a-rag-investment-assistant-7trsiwger#faq","mainEntity":[{"@type":"Question","name":"How can table data be chunked for RAG without losing headers and numeric context?","acceptedAnswer":{"@type":"Answer","text":"Instead of naively splitting large tables across chunks (which loses headers and the relationships that make numbers meaningful), GPT-4o-mini can convert table rows into self-contained text blocks that carry their headers, units, currencies, title, footnotes, and surrounding context. This approach raised retrieval accuracy on table-related questions from about 70% to about 94%, and pushed chart-related question accuracy to about 95%. daily.dev surfaces practical RAG engineering techniques like this for teams tuning retrieval accuracy."}},{"@type":"Question","name":"Why would an Airflow deployment stay unhealthy for weeks without anyone noticing?","acceptedAnswer":{"@type":"Answer","text":"A worker restart triggered by an out-of-memory condition can leave a DAG stuck without the scheduler ever marking the task as failed, and if the health endpoint doesn't check for that specific failure mode, the outage goes unreported. One deployment stayed broken for over 20 days due to a known upstream Airflow bug (issue #55029) before StatsD-to-CloudWatch metrics, DAG run timeouts, and SNS-backed alerts were added to catch it. engineers hardening Airflow monitoring can find similar production incident writeups on daily.dev."}},{"@type":"Question","name":"How can you combine vector search and keyword search for hybrid retrieval in a RAG system?","acceptedAnswer":{"@type":"Answer","text":"Pair a vector database like Pinecone for semantic ('soft') queries with a search engine like OpenSearch enforcing 'strict' must-contain, exclude, and filter conditions, since keyword and metadata filtering catches what vector similarity misses. In one production query, 500 Pinecone hits were narrowed down to just 8 relevant documents after OpenSearch filtering was applied. developers designing hybrid retrieval pipelines can compare real-world RAG architectures on daily.dev."}}]} ```