--- title: Best Crawling posts of 2021 | daily.dev description: The most upvoted Crawling posts of 2021, curated by the daily.dev community. canonical: https://daily.dev/tags/crawling/best-of/2021 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/tags/crawling/best-of/2021 og:type: website og:site_name: daily.dev og:title: Best Crawling posts of 2021 | daily.dev og:description: The most upvoted Crawling posts of 2021, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Crawling — 2021 1. 1 [](/posts/automa---a-chrome-extension-for-automating-your-browser-8siukuia5 "Automa - A chrome extension for automating your browser") Article Product Hunt · 5y Automa - A chrome extension for automating your browser An extension for automating your browser by connecting blocks. From auto-fill forms, to scraping data of the website, it's up to you what you want to do with this extension. It's free to download and use, and there are no plans to stop using it. 143 13 2. 2 [](/posts/how-to-build-a-web-scraper-with-node-js-htlrwnscc "How to Build a Web Scraper With Node.JS") Article DevDojo · 5y How to Build a Web Scraper With Node.JS This tutorial will demonstrate how to automate scraping data off of the website and using it for whatever purpose. The project will be built with Node.js and Express. The source code of the completed project can be found on CodeSandbox. Fork, tweak the scripts and run the code. 74 6 3. 3 [](/posts/how-to-build-an-amazon-product-scraper-with-node-js-smashing-magazine-mh-kreui3 "How To Build An Amazon Product Scraper With Node.js — Smashing Magazine") Article Smashing Magazine · 5y How To Build An Amazon Product Scraper With Node.js — Smashing Magazine Web scraping is the practice of extracting large amounts of web data through the use of software. Scraping Amazon product pages can net you invaluable data such as features, prices and reviews. While the act of scraping publicly available data is legal, Amazon has some measures to prevent it on their pages. 24 4. 4 [](/posts/is-web-scraping-legal--p61v-fam "Is Web Scraping Legal?") Article DZone · 5y Is Web Scraping Legal? The stance of the internet law on the legality of web scraping remains yet controversial. Most of the past legal face-offs between companies on web scraping ended up leaving traces of mind puzzles. With the court twists involved, a claimant might even end up guilty despite suing others for scraping their website. 21 1 5. 5 [](/posts/how-to-scrape-websites-with-node-js-and-cheerio-mtpd-gx-d "How to Scrape Websites with Node.js and Cheerio") Article freeCodeCamp · 5y How to Scrape Websites with Node.js and Cheerio .com. How to Scrape Websites with Node.js and Cheerio. Joseph Mawa explains how to scrape websites with the two tools. He says it's easier than it looks and it's faster than you think. He also shows you how to use Cheerios to scrape sites with the same code. 18 2 6. 6 [](/posts/developing-a-web-crawler-with-node-js-3-vrlnkfu "Developing a Web Crawler with Node.js") Article DZone · 5y Developing a Web Crawler with Node.js For business owners, all the data publicly available on the internet can be beneficial. This data can be helpful to generate leads for service providers and reach a bigger audience. Along with that, the data can also be used for the deep learning project to train the model. To run this code, you need to install a node.js package called a crawler. 14 7. 7 [](/posts/web-scraping-101-introduction-to-puppeteer-s1znyo1sw "Web Scraping 101: Introduction to Puppeteer") Article JavaScript in Plain English · 5y Web Scraping 101: Introduction to Puppeteer Puppeteer is a Node library that provides users an easy and accessible API to control Chrome or Chromium without an actual browser window (headless) We will write some code to perform login, data scraping, and pagination with Puppeteer on http://quotes.toscrape.com. 14 2 8. 8 [](/posts/building-a-web-scraper-in-go-with-colly-pcwapgfpj "Building a web scraper in Go with Colly") Article LogRocket · 5y Building a web scraper in Go with Colly In this tutorial, we will be taking a look at a Go package that allows us to build web scrapers, Colly. We will be building a basic web scraper that gets product information from an ecommerce store and saves the data to a JSON file. To follow along with the tutorial, you need to have Go installed on your local machine and you need at least a basic knowledge of Go. 13 9. 9 [](/posts/how-to-bypass-captchas-with-javascript-nzsx1keei "How to Bypass CAPTCHAs with JavaScript") Article JavaScript in Plain English · 5y How to Bypass CAPTCHAs with JavaScript How to Bypass CAPTCHAs with JavaScript, Puppeteer and 2Captcha. We will be automating the process of creating a new Reddit account, and in doing so we will be bypassing the common ‘I’m not a robot’ CAPTCHA. 12 1 10. 10 [](/posts/web-scraping-with-selenium-in-python-qaoozufgw "Web Scraping with Selenium in Python") Article Python in Plain English · 5y Web Scraping with Selenium in Python Learn to identify elements to get content, take screenshots and run JavaScript. Change the size of the window and capture screenshots. To prevent blockages, use proxies and special headers. With our headless browser, you can do all of that and more. Python3 is required for the code to function. 10 11. 11 [](/posts/the-easiest-way-to-build-a-web-scraper-using-javascript-and-node-js-skbcd6t8f "The Easiest Way to Build a Web Scraper Using JavaScript and Node.js") Article JavaScript in Plain English · 5y The Easiest Way to Build a Web Scraper Using JavaScript and Node.js The Easiest Way to Build a Web Scraper Using JavaScript and Node.js Dan Suciu shares his tips on building a web scraper using NodeJS. In the following section of this article, I will help you write your own web scraper application using Nodejs. 10 12. 12 [](/posts/pdf-generation-in-node-js-xlahui87n "PDF Generation in Node.js") Article JavaScript in Plain English · 5y PDF Generation in Node.js Puppeteer is a Node library that provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Using a puppeteer you can render the document and then generate PDF File from it. PDF make is a lightweight pure javascript library used to generate pdf. 10 [See all Crawling archives](/tags/crawling/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/tags/crawling/best-of/2021#page","url":"https://daily.dev/tags/crawling/best-of/2021","name":"Best Crawling Posts of 2021","description":"The most upvoted Crawling posts of 2021, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/tags/crawling/best-of/2021#items","numberOfItems":12,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/automa---a-chrome-extension-for-automating-your-browser-8siukuia5","name":"Automa - A chrome extension for automating your browser"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/how-to-build-a-web-scraper-with-node-js-htlrwnscc","name":"How to Build a Web Scraper With Node.JS"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/how-to-build-an-amazon-product-scraper-with-node-js-smashing-magazine-mh-kreui3","name":"How To Build An Amazon Product Scraper With Node.js — Smashing Magazine"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/is-web-scraping-legal--p61v-fam","name":"Is Web Scraping Legal?"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/how-to-scrape-websites-with-node-js-and-cheerio-mtpd-gx-d","name":"How to Scrape Websites with Node.js and Cheerio"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/developing-a-web-crawler-with-node-js-3-vrlnkfu","name":"Developing a Web Crawler with Node.js"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/web-scraping-101-introduction-to-puppeteer-s1znyo1sw","name":"Web Scraping 101: Introduction to Puppeteer"},{"@type":"ListItem","position":8,"url":"https://daily.dev/posts/building-a-web-scraper-in-go-with-colly-pcwapgfpj","name":"Building a web scraper in Go with Colly"},{"@type":"ListItem","position":9,"url":"https://daily.dev/posts/how-to-bypass-captchas-with-javascript-nzsx1keei","name":"How to Bypass CAPTCHAs with JavaScript"},{"@type":"ListItem","position":10,"url":"https://daily.dev/posts/web-scraping-with-selenium-in-python-qaoozufgw","name":"Web Scraping with Selenium in Python"},{"@type":"ListItem","position":11,"url":"https://daily.dev/posts/the-easiest-way-to-build-a-web-scraper-using-javascript-and-node-js-skbcd6t8f","name":"The Easiest Way to Build a Web Scraper Using JavaScript and Node.js"},{"@type":"ListItem","position":12,"url":"https://daily.dev/posts/pdf-generation-in-node-js-xlahui87n","name":"PDF Generation in Node.js"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tags","item":"https://daily.dev/tags"},{"@type":"ListItem","position":3,"name":"Crawling","item":"https://daily.dev/tags/crawling"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/tags/crawling/best-of"},{"@type":"ListItem","position":5,"name":"2021"}]}]} ```