<blockquote>
<a href="http://WebCrawler.buzz" target="_blank" rel="noopener nofollow">WebCrawler.buzz</a> is a free domain crawler that discovers every page on any website and extracts rich SEO data. Paste a URL, hit crawl, get a full report. No signup, no page limits.
</blockquote>
Built with Express.js v5, TypeScript, BullMQ, Redis, and Neon PostgreSQL. The server creates a job and pushes it to a BullMQ queue. A BFS crawler processes pages in the background. You get a shareable tracking URL instantly with real-time progress polling every 5 seconds.
<ul>
<li>Data per page: title, meta description, HTTP status, response time, content type, page size, internal/external link counts, redirects, and indexability.</li>
<li>Key features: email notifications, CSV export via email, duplicate domain detection, auto-cleanup of data, Swagger API docs, health check, and public stats endpoints.</li>
<li>Tech stack: Node.js, TypeScript, Express.js v5, Neon PostgreSQL, BullMQ + Redis, Axios + Cheerio, Nodemailer, Tailwind CSS.</li>
</ul>
MIT licensed and live at <a href="http://webcrawler.buzz" target="_blank" rel="noopener nofollow">webcrawler.buzz</a>.

Hasibul Islam

WebCrawler.buzz is a free, no-signup web crawler that discovers all pages on any website and extracts SEO data including titles, meta descriptions, HTTP status, response times, and link counts. Built with Express.js v5, TypeScript, BullMQ, Redis, and Neon PostgreSQL, it uses a BFS crawling algorithm with background job processing via queues. Features include real-time progress polling, email notifications, CSV export, duplicate domain detection, Swagger API docs, and public stats endpoints. MIT licensed and publicly available.

webcrawler.buzz: Free Domain Crawler