--- title: How I built a 500k-Domain Search Engine for Makers in a... description: A solo weekend project built a 500k-domain personal search engine focused on surfacing indie portfolios, zines, and hobby projects instead of corporate SEO... canonical: https://daily.dev/posts/how-i-built-a-500k-domain-search-engine-for-makers-in-a-weekend-for-10-lbhwfd3kg twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How I built a 500k-Domain Search Engine for Makers in a Weekend for $10 | daily.dev og:description: A solo weekend project built a 500k-domain personal search engine focused on surfacing indie portfolios, zines, and hobby projects instead of corporate SEO... og:url: https://daily.dev/posts/how-i-built-a-500k-domain-search-engine-for-makers-in-a-weekend-for-10-lbhwfd3kg og:image: https://api.daily.dev/og/posts/LbHWfD3kG.png og:image:alt: How I built a 500k-Domain Search Engine for Makers in a Weekend for $10 og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How I built a 500k-Domain Search Engine for Makers in a Weekend for $10 **[Hacker News](https://daily.dev/sources/hn)** · 12 min read · 1 upvotes · 0 comments ## Summary A solo weekend project built a 500k-domain personal search engine focused on surfacing indie portfolios, zines, and hobby projects instead of corporate SEO sludge. The build used a custom crawler, a small local/rented LLM to summarize homepages, and a priority-weighted queue to steer discovery away from docs and ecommerce sites toward niche communities. Total GPU rental cost was about $10, with detailed notes on throughput tuning, subdomain capping, category taxonomy problems, and an automated 'spiral detector' that blocked spammy domains without human babysitting. The code is being released as open source (Marlin) though the author isn't releasing the compiled database. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much does it cost to crawl and summarize a large number of websites using a rented GPU and a small language model? Roughly a dollar per 100,000 sites summarized, using a rented GPU running an open source inference server at about thirty five cents per hour. Sustained throughput reached around 600 summaries per minute on a workstation-class GPU with a dedicated, unshared set of CPU cores, making a million-site index cost around ten dollars total. _Anyone estimating the cost of an LLM-powered crawl project can compare notes like these on daily.dev before committing a GPU budget._ ### Why would a web crawler using an LLM to summarize pages hallucinate the wrong topic for a page? A near-empty domain-parking page with the word 'furry' in its hostname was summarized by a language model as being about the furry community, purely from guessing based on the domain name rather than actual page content. The fix was prioritizing visible page text over the title, the title over the domain name, and skipping the model entirely for near-empty or parked pages. _Developers building LLM summarization pipelines can trade war stories on prompt failure modes like this via daily.dev._ ### How do you stop a handful of blog platforms like Tumblr or Neocities from dominating a web crawl? Cap subdomains per root domain rather than blocking the platform outright: once a root domain produces more than 100 subdomains, stop enqueueing new ones from it, while still always allowing the main domain. Applying this rule once instantly removed over 45,000 queued pages, though already-queued Tumblr pages still contributed over 9,000 pages to the final index. _Crawl-tuning tricks like subdomain capping are the kind of practical detail worth surfacing on daily.dev for anyone scaling their own scraper._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 52 comments across hackernews (as of 2026-09-14). **TL;DR:** Discussion splits sharply between people dismissing the write-up as AI-generated 'slop' not worth reading and others who find the underlying project (a personal, steerable search engine focused on niche sites) genuinely interesting; there's also technical debate about domain discovery, scraping limitations, and what the tool is actually for. **Sentiment:** 25% positive · 30% mixed · 45% skeptical **The case for** - Some found real value or inspiration in the write-up despite calling parts of it 'slop', saying it doesn't need to be read closely to be useful. - Several see personal/homebrew search as an interesting, underexplored space now that compute for crawling and summarizing is cheap. - People noted the priority-steering approach lets you focus compute on a curated slice of the web efficiently. - One commenter with search-engine experience called auto-labeled directories a legitimately interesting angle on a real problem (web discovery). **The pushback** - Many commenters felt the article itself reads as AI-generated and low-effort, making it hard or not worth their time to parse. - Skeptics argued the scraping approach silently skips anything behind JavaScript, captchas, or Cloudflare, so the resulting data isn't very representative or practically useful. - Some questioned the practical utility of the tool, saying it won't replace existing search engines like Google or DuckDuckGo. - A few pushed back on the 'weekend' framing, saying the amount of work evident contradicts the low-effort branding. - One long tangent devolved into a heated argument about AI-generated writing and offloading critical thinking, unrelated to the project's technical merits. **By community** - hackernews (heated): A loud argument over whether the write-up is AI 'slop' worth reading dominates, while a separate, calmer thread digs into the crawler's technical tradeoffs and the broader idea of personal search engines. **Hottest debate:** Whether the AI-assisted, possibly AI-written article should be dismissed as low-value 'slop' regardless of the project's underlying merit. **Open questions** - How is the initial seed list of domains built, since the repo doesn't make this obvious? - How much of the 'personal web' is actually reachable given the crawler skips anything behind bot protections or JS rendering? - Is there real practical use for a 500k-domain personal search engine beyond a fun side project? **Highlights** > Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. 5. `code is going up as open source` soon (TM) > — [iFire on hackernews · 3 comments](https://news.ycombinator.com/item?id=49286510) > From what I understand the aim was not to collect all the domains on the web but focus on personal website, etc. and avoid corporate web sites. > — [BaudouinVH on hackernews](https://news.ycombinator.com/item?id=49288545) > Thats the fun part, the user just went with happy path. Javascript, captchas, cloudflare protected content did not made to the catalogue. This sort of use case exists in LLM training data a lot which makes it easier. The data gathered by the user is not really practically useful cause there are way too many gotchas when it comes to web scraping and building a catalogue (source: I have done scraping for a particular domain data and had to do at least 10+ iterations to get it >90 right) > — [sandeepkd on hackernews](https://news.ycombinator.com/item?id=49287727) > To be fair they are a supremely interesting problem to hack away at, and one that will meet you where you are. Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence. Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing. It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later. > — [marginalia\_nu on hackernews](https://news.ycombinator.com/item?id=49288934) > It's not for that, sorry, I should have been more specific. It's for people who wanna put in the effort and steer their own crawl to surface their own slice of the web. The article is just a little story of the journey > — [dreamforever on hackernews](https://news.ycombinator.com/item?id=49289568) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49285718) · 87 points · 52 comments ## Similar posts on daily.dev - [Cloudflare AI Search: give your agents a search engine for your data](https://daily.dev/posts/cloudflare-ai-search-give-your-agents-a-search-engine-for-your-data-ctstzert3) · Cloudflare · 1 upvotes · 0 comments - [Search Quality Assurance with AI as a Judge](https://daily.dev/posts/search-quality-assurance-with-ai-as-a-judge-ckpklk2ko) · Zalando · 3 upvotes · 0 comments - [My local LLM's real problem wasn't hallucination, it was a two-year blind spot](https://daily.dev/posts/my-local-llm-s-real-problem-wasn-t-hallucination-it-was-a-two-year-blind-spot-eqscg8rag) · XDA Developers · 0 upvotes · 0 comments - [Give Google the boot by building your own search engine](https://daily.dev/posts/give-google-the-boot-by-building-your-own-search-engine-ybiewldgg) · The Register · 0 upvotes · 0 comments --- Tags: [#startup](https://daily.dev/tags/startup), [#llm](https://daily.dev/tags/llm), [#gpu](https://daily.dev/tags/gpu) [View this post on daily.dev](https://daily.dev/posts/how-i-built-a-500k-domain-search-engine-for-makers-in-a-weekend-for-10-lbhwfd3kg) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How I built a 500k-Domain Search Engine for Makers in a Weekend for $10","url":"https://daily.dev/posts/how-i-built-a-500k-domain-search-engine-for-makers-in-a-weekend-for-10-lbhwfd3kg","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-i-built-a-500k-domain-search-engine-for-makers-in-a-weekend-for-10-lbhwfd3kg"},"datePublished":"2026-08-13T15:25:05.198Z","dateModified":"2026-09-14T06:41:41.139Z","description":"A solo weekend project built a 500k-domain personal search engine focused on surfacing indie portfolios, zines, and hobby projects instead of corporate SEO...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/143a2a263481d775c7067df97ccfb751?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/143a2a263481d775c7067df97ccfb751?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-i-built-a-500k-domain-search-engine-for-makers-in-a-weekend-for-10-lbhwfd3kg","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"startup,llm,gpu","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"How I built a 500k-Domain Search Engine for Makers in a Weekend for $10"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-i-built-a-500k-domain-search-engine-for-makers-in-a-weekend-for-10-lbhwfd3kg#faq","mainEntity":[{"@type":"Question","name":"How much does it cost to crawl and summarize a large number of websites using a rented GPU and a small language model?","acceptedAnswer":{"@type":"Answer","text":"Roughly a dollar per 100,000 sites summarized, using a rented GPU running an open source inference server at about thirty five cents per hour. Sustained throughput reached around 600 summaries per minute on a workstation-class GPU with a dedicated, unshared set of CPU cores, making a million-site index cost around ten dollars total. Anyone estimating the cost of an LLM-powered crawl project can compare notes like these on daily.dev before committing a GPU budget."}},{"@type":"Question","name":"Why would a web crawler using an LLM to summarize pages hallucinate the wrong topic for a page?","acceptedAnswer":{"@type":"Answer","text":"A near-empty domain-parking page with the word 'furry' in its hostname was summarized by a language model as being about the furry community, purely from guessing based on the domain name rather than actual page content. The fix was prioritizing visible page text over the title, the title over the domain name, and skipping the model entirely for near-empty or parked pages. Developers building LLM summarization pipelines can trade war stories on prompt failure modes like this via daily.dev."}},{"@type":"Question","name":"How do you stop a handful of blog platforms like Tumblr or Neocities from dominating a web crawl?","acceptedAnswer":{"@type":"Answer","text":"Cap subdomains per root domain rather than blocking the platform outright: once a root domain produces more than 100 subdomains, stop enqueueing new ones from it, while still always allowing the main domain. Applying this rule once instantly removed over 45,000 queued pages, though already-queued Tumblr pages still contributed over 9,000 pages to the final index. Crawl-tuning tricks like subdomain capping are the kind of practical detail worth surfacing on daily.dev for anyone scaling their own scraper."}}]} ```