--- title: Agents Need a New Kind of Web Search | daily.dev description: Traditional web search APIs return only URLs and snippets, forcing AI agents to fetch and clean pages themselves — a costly 'retrieval tax' that compounds... canonical: https://daily.dev/posts/agents-need-a-new-kind-of-web-search-wibepp98q twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Agents Need a New Kind of Web Search | daily.dev og:description: Traditional web search APIs return only URLs and snippets, forcing AI agents to fetch and clean pages themselves — a costly 'retrieval tax' that compounds... og:url: https://daily.dev/posts/agents-need-a-new-kind-of-web-search-wibepp98q og:image: https://api.daily.dev/og/posts/WIBEPP98Q.png og:image:alt: Agents Need a New Kind of Web Search og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Agents Need a New Kind of Web Search **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 11 min read · 1 upvotes · 0 comments ## Summary Traditional web search APIs return only URLs and snippets, forcing AI agents to fetch and clean pages themselves — a costly 'retrieval tax' that compounds across multi-hop loops. A token-counting experiment shows that a naive 3-hop search loop costs ~28,700 tokens versus ~6,900 for a pre-indexed owned index. The post argues agents need search tools that return finished, structured documents rather than pointers to content. It introduces Seltz, a web index with People, News, and Wiki scopes that returns complete records in a single API call. The piece also briefly covers the evolution from RAG to Agentic RAG to AI Memory, noting that read-write memory enables continual learning but introduces new challenges like memory corruption and forgetting strategies. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does giving an AI agent a web search tool result in high token costs? Because typical search APIs return only links and short snippets, not full content, forcing the agent to fetch raw HTML, strip markup, and dig out usable text on every call. This repeated fetch-and-clean overhead, called the retrieval tax, compounds across multiple search hops since each hop re-bills the growing context window carrying pages already read. _Anyone tuning agent search costs can track retrieval architecture comparisons like this on daily.dev._ ### How much more expensive is a multi-hop web search loop compared to using a pre-indexed search service for an agent? A three-hop web search loop cost roughly 28,700 tokens versus about 6,900 tokens for a single call to an owned index that pre-crawls and cleans pages, over four times more expensive. Answering the same question purely from the model's memory, with zero retrieval, cost only about 600 tokens as a baseline. _Developers weighing search architectures for agent pipelines can follow cost breakdowns like this on daily.dev._ ### What is the difference between a SERP, a neural search tool, and an owned index for agent retrieval? A SERP returns plain URLs and snippets, leaving the agent to fetch and clean pages itself. A neural search tool returns highlight fragments that identify the right content but still require another call for the full record. An owned index pre-crawls and structures pages so one query returns a complete, ready-to-use document, cutting the extra retrieval steps entirely. _Teams designing agent retrieval pipelines can compare these approaches through daily.dev._ ## Similar posts on daily.dev - [Your agent wants to search like a 2010 quant](https://daily.dev/posts/your-agent-wants-to-search-like-a-2010-quant-lx3kju83b) · Vespa Blog · 1 upvotes · 0 comments - [Power Up Your AI Agent With Live Web Search, for Fewer Tokens](https://daily.dev/posts/power-up-your-ai-agent-with-live-web-search-for-fewer-tokens-tksh6izca) · SitePoint · 1 upvotes · 0 comments - [Agentic RAG: Let the Agent Search](https://daily.dev/posts/agentic-rag-let-the-agent-search-fbdgqwlmi) · Towards Data Science · 3 upvotes · 1 comments - [AI Search: the search primitive for your agents](https://daily.dev/posts/ai-search-the-search-primitive-for-your-agents-esbieslcq) · Cloudflare · 1 upvotes · 0 comments - [AI teams are spending months on web scrapers that SerpApi replaces with one API call](https://daily.dev/posts/ai-teams-are-spending-months-on-web-scrapers-that-serpapi-replaces-with-one-api-call-pjoalfsh4) · The New Stack · 1 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#rag](https://daily.dev/tags/rag) [View this post on daily.dev](https://daily.dev/posts/agents-need-a-new-kind-of-web-search-wibepp98q) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Agents Need a New Kind of Web Search","url":"https://daily.dev/posts/agents-need-a-new-kind-of-web-search-wibepp98q","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/agents-need-a-new-kind-of-web-search-wibepp98q"},"datePublished":"2026-07-16T20:38:25.292Z","dateModified":"2026-09-14T07:50:35.865Z","description":"Traditional web search APIs return only URLs and snippets, forcing AI agents to fetch and clean pages themselves — a costly 'retrieval tax' that compounds...","image":"https://substackcdn.com/image/fetch/$s_!hMNL!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd299fa60-2f68-4efd-8e04-f8466e001faf_976x750.gif","thumbnailUrl":"https://substackcdn.com/image/fetch/$s_!hMNL!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd299fa60-2f68-4efd-8e04-f8466e001faf_976x750.gif","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/agents-need-a-new-kind-of-web-search-wibepp98q","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,ai-agents,rag","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Agents Need a New Kind of Web Search"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/agents-need-a-new-kind-of-web-search-wibepp98q#faq","mainEntity":[{"@type":"Question","name":"Why does giving an AI agent a web search tool result in high token costs?","acceptedAnswer":{"@type":"Answer","text":"Because typical search APIs return only links and short snippets, not full content, forcing the agent to fetch raw HTML, strip markup, and dig out usable text on every call. This repeated fetch-and-clean overhead, called the retrieval tax, compounds across multiple search hops since each hop re-bills the growing context window carrying pages already read. Anyone tuning agent search costs can track retrieval architecture comparisons like this on daily.dev."}},{"@type":"Question","name":"How much more expensive is a multi-hop web search loop compared to using a pre-indexed search service for an agent?","acceptedAnswer":{"@type":"Answer","text":"A three-hop web search loop cost roughly 28,700 tokens versus about 6,900 tokens for a single call to an owned index that pre-crawls and cleans pages, over four times more expensive. Answering the same question purely from the model's memory, with zero retrieval, cost only about 600 tokens as a baseline. Developers weighing search architectures for agent pipelines can follow cost breakdowns like this on daily.dev."}},{"@type":"Question","name":"What is the difference between a SERP, a neural search tool, and an owned index for agent retrieval?","acceptedAnswer":{"@type":"Answer","text":"A SERP returns plain URLs and snippets, leaving the agent to fetch and clean pages itself. A neural search tool returns highlight fragments that identify the right content but still require another call for the full record. An owned index pre-crawls and structures pages so one query returns a complete, ready-to-use document, cutting the extra retrieval steps entirely. Teams designing agent retrieval pipelines can compare these approaches through daily.dev."}}]} ```