--- title: Best Towards AI posts — September 2024 | daily.dev description: The most upvoted Towards AI posts from September 2024, curated by the daily.dev community. canonical: https://daily.dev/sources/tai/best-of/2024/09 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/sources/tai/best-of/2024/09 og:type: website og:site_name: daily.dev og:title: Best Towards AI posts — September 2024 | daily.dev og:description: The most upvoted Towards AI posts from September 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Towards AI — September 2024 1. 1 [](https://daily.dev/posts/retrieval-interleaved-generation-rig-when-real-time-data-retrieval-meets-response-generation-hdbgpcqvk "Retrieval Interleaved Generation (RIG): When real-time data retrieval meets response generation") Article Towards AI · 2y Retrieval Interleaved Generation (RIG): When real-time data retrieval meets response generation Retrieval Interleaved Generation (RIG) is a cutting-edge technique in natural language processing that dynamically combines real-time data retrieval with response generation. Unlike Retrieval-Augmented Generation (RAG), which performs a single retrieval step before generating a response, RIG interleaves multiple retrievals during the response creation process, ensuring up-to-date and accurate information. This method significantly reduces hallucinations and improves accuracy for complex, data-dependent queries. RIG is particularly useful in fields like healthcare, finance, and scientific research, where real-time, precise information is crucial. 32 1 2. 2 [](https://daily.dev/posts/teaching-rag-to-remember-how-memorag-enhances-question-answering-through-memory-teagm4erv "Teaching RAG to “Remember”: How MemoRAG Enhances Question-Answering Through Memory") Article Towards AI · 2y Teaching RAG to “Remember”: How MemoRAG Enhances Question-Answering Through Memory MemoRAG introduces a long-term memory system to enhance retrieval efficiency and address limitations of traditional RAG systems in handling complex or ambiguous information needs. Through code analysis, training process details, and case study evaluation, MemoRAG demonstrates improved performance by recalling relevant information based on context. 24 3. 3 [](https://daily.dev/posts/why-openai-s-o1-model-is-a-scam-63a8kt779 "Why OpenAI’s o1 Model Is A Scam") Article Towards AI · 2y Why OpenAI’s o1 Model Is A Scam OpenAI's o1 model claims to advance AI by making it think before responding, using the Chain of Thought (CoT) technique. However, the author argues that the model is mostly a repackaged marketing ploy, as CoT has been around for years. The post includes a Python implementation of CoT and discusses the potential benefits of OpenAI's reinforcement learning for better intermediate step performance. Readers are advised to critically evaluate such new features before committing financially. 21 3 4. 4 [](https://daily.dev/posts/a-practical-approach-to-using-web-data-for-ai-and-llms-ivchhxqgq "A Practical Approach to Using Web Data for AI and LLMs") Article Towards AI · 2y A Practical Approach to Using Web Data for AI and LLMs Businesses and researchers are increasingly relying on high-quality web data for AI and large language models (LLMs). Bright Data offers advanced tools to collect, manage, and use this data, making it easier to train models, improve real-time applications, and perform sentiment analysis. Their solutions ensure ethical data collection and compliance with privacy regulations, providing scalable infrastructure to handle various project needs. This is crucial as AI development demands not just vast amounts of data but also quality and relevance. 20 5. 5 [](https://daily.dev/posts/building-a-conversational-medical-form-with-gpt-4-langchain-and-sqlite-tr27vxnlz "Building a Conversational Medical Form with GPT-4, Langchain, and SQLite") Article Towards AI · 2y Building a Conversational Medical Form with GPT-4, Langchain, and SQLite Imagine replacing traditional static medical forms with a conversational assistant that uses GPT-4, guided by Langchain, to understand and store patient symptoms in a SQLite database. This post guides you through creating an interactive AI-driven medical form that alleviates patient stress by simulating a more personalized and caring interaction. 20 6. 6 [](https://daily.dev/posts/transformer-architecture-part--1-vnk1zuca4 "Transformer Architecture Part -1") Article Towards AI · 2y Transformer Architecture Part -1 Transformers have revolutionized deep learning, excelling in language and vision tasks. The core architecture consists of identical encoder and decoder blocks, each featuring self-attention, feed-forward neural networks, add & norm layers, and residual connections. The process begins with tokenization, text vectorization, and positional encoding. Multi-head attention then contextualizes these vectors, followed by normalization and passing through feed-forward networks. The architecture ensures efficient handling of complex data patterns while maintaining consistent dimensionality for smooth training. 20 7. 7 [](https://daily.dev/posts/building-deploying-a-fastapi-video-description-app-from-code-to-cloud-with-gpt-docker-and-azure-uuy2qw2hi "Building & Deploying a FastAPI Video Description App: From Code to Cloud with GPT, Docker and Azure") Article Towards AI · 2y Building & Deploying a FastAPI Video Description App: From Code to Cloud with GPT, Docker and Azure Learn how to build a FastAPI application that converts video URLs into text descriptions using AI. This beginner-friendly guide covers setting up a Python virtual environment, containerizing the app with Docker, and deploying it to Azure Web Apps. By the end, you'll have a fully functioning app deployed online. 19 8. 8 [](https://daily.dev/posts/mathematical-transformations-in-feature-engineering-log-reciprocal-and-power-transforms-explained-tsgvmpzjn "Mathematical Transformations in Feature Engineering: Log, Reciprocal, and Power Transforms Explained with Visualization") Article Towards AI · 2y Mathematical Transformations in Feature Engineering: Log, Reciprocal, and Power Transforms Explained with Visualization Learn how Log, Reciprocal, and Power Transforms can normalize your dataset, making it more manageable and improving the performance of machine learning models. Discover the reasons for data transformation and how to visualize the impacts using Python libraries like QQPlot and distplot. 15 9. 9 [](https://daily.dev/posts/building-a-smart-chatbot-with-openai-and-pinecone-a-simple-guide-vcwyuhslz "Building a Smart Chatbot with OpenAI and Pinecone: A Simple Guide") Article Towards AI · 2y Building a Smart Chatbot with OpenAI and Pinecone: A Simple Guide Learn how to build a simple Retrieval-Augmented Generation (RAG) chatbot using Python, incorporating Pinecone for the vector database and OpenAI for the language model. This guide addresses the issue of chatbot hallucinations by enhancing the language model with external knowledge sources. Step-by-step instructions include setting up API keys, installing necessary packages, creating a Pinecone index, chunking and embedding documents, and querying Pinecone for accurate responses. 13 10. 10 [](https://daily.dev/posts/build-your-own-ai-research-assistant-vnoezj9vu "Build Your Own AI Research Assistant") Article Towards AI · 2y Build Your Own AI Research Assistant Discover how to build an AI research assistant using LangChain, Gradio, and arXiv integration. Set up your environment, install relevant libraries, and understand the purpose of each tool. Learn to download and process research papers, embed texts, and create efficient retrieval systems with Qdrant. Finally, create an interactive web interface using Gradio to make your assistant accessible. 13 11. 11 [](https://daily.dev/posts/reliable-agentic-rag-with-llm-trustworthiness-estimates-6cymnlq7n "Reliable Agentic RAG with LLM Trustworthiness Estimates") Article Towards AI · 2y Reliable Agentic RAG with LLM Trustworthiness Estimates An agentic system is introduced to enhance the reliability of answers in Retrieval-Augmented Generation (RAG), while managing latency and compute costs. The system uses trustworthiness scores for LLM outputs to dynamically adjust retrieval strategies until a trustworthy response is generated. Employing an Uncertainty Estimator, the system can identify and improve untrustworthy responses by escalating retrieval methods. Trustworthy answers are ensured by starting with cheaper retrieval strategies and escalating the complexity only when necessary. 12 1 12. 12 [](https://daily.dev/posts/38-back-to-basics-rag-transformers-ml-optimization-and-llm-evaluation--i4q1pdkhp "#38 Back to Basics — RAG, Transformers, ML Optimization, and LLM Evaluation.") Article Towards AI · 2y #38 Back to Basics — RAG, Transformers, ML Optimization, and LLM Evaluation. The post delves into the relevance of RAG (Retrieval-Augmented Generation), comparing it against models like Gemini that process millions of tokens. It highlights why RAG will remain useful for specific applications. There's a mention of a free masterclass on AI tools, a project spotlight on an AI-driven job search assistant, and various collaboration opportunities in the AI community. A featurette on a Streamlit app for RAG evaluation and discussions on the importance of transformer architecture in NLP and querying SQL databases using LLM agents are also included. 11 13. 13 [](https://daily.dev/posts/journey-from-data-warehouse-to-lake-to-lakehouse-ifah93kj7 "Journey From Data Warehouse To Lake To Lakehouse") Article Towards AI · 2y Journey From Data Warehouse To Lake To Lakehouse The post provides a fictional story to simplify the understanding of data storage concepts such as Data Warehouse, Data Lake, and Data Lakehouse. It highlights the evolution from the structured data storage of Data Warehouses, to the flexible, low-cost storage of Data Lakes, and finally to the comprehensive and efficient storage solutions of Data Lakehouses, which combine the benefits of both previous systems. Key concepts like schema-on-read and schema-on-write are explained, and top providers for each storage solution are recommended. 10 14. 14 [](https://daily.dev/posts/get-the-most-out-of-llama-3-1-oee9afq2z "Get The Most Out of Llama 3.1") Article Towards AI · 2y Get The Most Out of Llama 3.1 Llama 3.1, the first open model with nearly half a trillion parameters, introduces critical advancements in preprocessing, training configuration, and model alignment. Emphasizing the removal of toxic and redundant data, domain balancing, and gradual increase in batch size and sequence length, it aims for stability and computational efficiency. Annotations are refined for quality, and DPO is preferred over PPO for model alignment. Post-training, the model is fine-tuned for expertise in code, multilingual capabilities, and math reasoning, ensuring it only answers questions it is confident about. 10 1 [See all Towards AI archives](/sources/tai/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/sources/tai/best-of/2024/09#page","url":"https://daily.dev/sources/tai/best-of/2024/09","name":"Best Towards AI Posts — September 2024","description":"The most upvoted Towards AI posts from September 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/sources/tai/best-of/2024/09#items","numberOfItems":14,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/retrieval-interleaved-generation-rig-when-real-time-data-retrieval-meets-response-generation-hdbgpcqvk","name":"Retrieval Interleaved Generation (RIG): When real-time data retrieval meets response generation"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/teaching-rag-to-remember-how-memorag-enhances-question-answering-through-memory-teagm4erv","name":"Teaching RAG to “Remember”: How MemoRAG Enhances Question-Answering Through Memory"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/why-openai-s-o1-model-is-a-scam-63a8kt779","name":"Why OpenAI’s o1 Model Is A Scam"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/a-practical-approach-to-using-web-data-for-ai-and-llms-ivchhxqgq","name":"A Practical Approach to Using Web Data for AI and LLMs"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/building-a-conversational-medical-form-with-gpt-4-langchain-and-sqlite-tr27vxnlz","name":"Building a Conversational Medical Form with GPT-4, Langchain, and SQLite"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/transformer-architecture-part--1-vnk1zuca4","name":"Transformer Architecture Part -1"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/building-deploying-a-fastapi-video-description-app-from-code-to-cloud-with-gpt-docker-and-azure-uuy2qw2hi","name":"Building & Deploying a FastAPI Video Description App: From Code to Cloud with GPT, Docker and Azure"},{"@type":"ListItem","position":8,"url":"https://daily.dev/posts/mathematical-transformations-in-feature-engineering-log-reciprocal-and-power-transforms-explained-tsgvmpzjn","name":"Mathematical Transformations in Feature Engineering: Log, Reciprocal, and Power Transforms Explained with Visualization"},{"@type":"ListItem","position":9,"url":"https://daily.dev/posts/building-a-smart-chatbot-with-openai-and-pinecone-a-simple-guide-vcwyuhslz","name":"Building a Smart Chatbot with OpenAI and Pinecone: A Simple Guide"},{"@type":"ListItem","position":10,"url":"https://daily.dev/posts/build-your-own-ai-research-assistant-vnoezj9vu","name":"Build Your Own AI Research Assistant"},{"@type":"ListItem","position":11,"url":"https://daily.dev/posts/reliable-agentic-rag-with-llm-trustworthiness-estimates-6cymnlq7n","name":"Reliable Agentic RAG with LLM Trustworthiness Estimates"},{"@type":"ListItem","position":12,"url":"https://daily.dev/posts/38-back-to-basics-rag-transformers-ml-optimization-and-llm-evaluation--i4q1pdkhp","name":"#38 Back to Basics — RAG, Transformers, ML Optimization, and LLM Evaluation."},{"@type":"ListItem","position":13,"url":"https://daily.dev/posts/journey-from-data-warehouse-to-lake-to-lakehouse-ifah93kj7","name":"Journey From Data Warehouse To Lake To Lakehouse"},{"@type":"ListItem","position":14,"url":"https://daily.dev/posts/get-the-most-out-of-llama-3-1-oee9afq2z","name":"Get The Most Out of Llama 3.1"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Sources","item":"https://daily.dev/sources"},{"@type":"ListItem","position":3,"name":"Towards AI","item":"https://daily.dev/sources/tai"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/sources/tai/best-of"},{"@type":"ListItem","position":5,"name":"September 2024"}]}]} ```