--- title: How Web Data Infrastructure Powers the Next Generation... description: A product manager at web scraping and proxy company Oxylabs recounts three internal case studies illustrating how AI's demand for fresh, real-time web data... canonical: https://daily.dev/posts/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-emaityt-oxylabs-m43hxkil1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs | daily.dev og:description: A product manager at web scraping and proxy company Oxylabs recounts three internal case studies illustrating how AI's demand for fresh, real-time web data... og:url: https://daily.dev/posts/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-emaityt-oxylabs-m43hxkil1 og:image: https://api.daily.dev/og/posts/m43HXKIl1.png og:image:alt: How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 19 min read · 0 upvotes · 0 comments ## Summary A product manager at web scraping and proxy company Oxylabs recounts three internal case studies illustrating how AI's demand for fresh, real-time web data reshaped the company's infrastructure. Stories include building a multimodal video API suite (downloader, transcripts, subtitles, metadata) under a two-week deadline, redesigning a search results (SERP) scraper from 4-second average latency down to 550ms to support AI retrieval pipelines, and scaling a web unblocker from 10,000 to 60,000 requests per second (now targeting 100,000+) while dealing with observability and load-testing challenges at scale. The talk frames adaptation, not raw model improvements, as the real driver of next-generation AI infrastructure. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [How web intelligence is powering the next wave of AI Infrastructure](https://daily.dev/posts/how-web-intelligence-is-powering-the-next-wave-of-ai-infrastructure-5pnyvqlnz) · The Next Web · 0 upvotes · 0 comments - [The emergence of the web data infrastructure layer for AI](https://daily.dev/posts/the-emergence-of-the-web-data-infrastructure-layer-for-ai-l8z7cc1y9) · MIT Technology Review · 0 upvotes · 0 comments - [How AI-Powered APIs and Data Extraction Are Reshaping Web Development](https://daily.dev/posts/how-ai-powered-apis-and-data-extraction-are-reshaping-web-development-yrxl51xvh) · SitePoint · 0 upvotes · 0 comments - [How to stop holding AI agents back](https://daily.dev/posts/how-to-stop-holding-ai-agents-back-86grinkth) · The Next Web · 0 upvotes · 0 comments - [Why AI models need a real-time web intelligence layer](https://daily.dev/posts/why-ai-models-need-a-real-time-web-intelligence-layer-kyp0zq485) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#big-data](https://daily.dev/tags/big-data), [#rag](https://daily.dev/tags/rag), [#crawling](https://daily.dev/tags/crawling) [View this post on daily.dev](https://daily.dev/posts/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-emaityt-oxylabs-m43hxkil1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs","url":"https://daily.dev/posts/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-emaityt-oxylabs-m43hxkil1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-emaityt-oxylabs-m43hxkil1"},"datePublished":"2026-08-14T17:03:22.101Z","dateModified":"2026-08-14T17:03:40.614Z","description":"A product manager at web scraping and proxy company Oxylabs recounts three internal case studies illustrating how AI's demand for fresh, real-time web data...","image":"https://i.ytimg.com/vi/1UmZHb_E_SM/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/1UmZHb_E_SM/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-emaityt-oxylabs-m43hxkil1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"big-data,rag,crawling","timeRequired":"PT19M","video":{"@type":"VideoObject","name":"How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs","description":"A product manager at web scraping and proxy company Oxylabs recounts three internal case studies illustrating how AI's demand for fresh, real-time web data...","thumbnailUrl":"https://i.ytimg.com/vi/1UmZHb_E_SM/sddefault.jpg","uploadDate":"2026-08-14T17:03:22.101Z","duration":"PT19M","url":"https://api.daily.dev/r/m43HXKIl1","embedUrl":"https://www.youtube.com/embed/1UmZHb_E_SM"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs"}]} ```