--- title: Best Crawling posts of 2024 | daily.dev description: The most upvoted Crawling posts of 2024, curated by the daily.dev community. canonical: https://daily.dev/tags/crawling/best-of/2024 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/tags/crawling/best-of/2024 og:type: website og:site_name: daily.dev og:title: Best Crawling posts of 2024 | daily.dev og:description: The most upvoted Crawling posts of 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Crawling — 2024 1. 1 [](/posts/using-gpt-4o-for-web-scraping-orrhtenf9 "Using GPT-4o for web scraping") Article Hacker News · 2y Using GPT-4o for web scraping A developer experimented with using GPT-4o's structured outputs for web scraping, creating an AI-assisted web scraper. While the model performed well with simple and complex tables, it struggled with combined rows and generating XPaths. Cost is a concern due to the model's character volume requirements. Future improvements could include better UX through capturing browser events and further refining HTML data cleanup. 565 15 2. 2 [](/posts/7-open-source-projects-you-should-know---python-edition--ot12y3hmc "7 Open Source Projects You Should Know - Python Edition ✔️") Article This is Learning · 2y 7 Open Source Projects You Should Know - Python Edition ✔️ Explore seven noteworthy open source projects written in Python, including pandas for data analysis, Apache Airflow for workflow management, G4F for decentralized AI technologies, Scrapy for web scraping, Ultroid as a Telegram UserBot, Zulip for team collaboration, and Freqtrade for crypto trading. Discover their features, installation guides, and more to enhance your coding endeavors. 366 7 3. 3 [](/posts/the-best-project-idea-to-learn-aws-yvitbmafy "The BEST Project Idea to Learn AWS") Video Be A Better Dev · 2y The BEST Project Idea to Learn AWS This post shares an engaging project idea to learn AWS hands-on, focusing on essential services like EC2, RDS, S3, IAM, VPC, and CloudWatch. The project involves building a web scraper to extract data from websites, providing practical skills that are valuable for a career in cloud engineering. The author emphasizes the importance of hands-on experience over certifications for solidifying AWS concepts. 184 1 4. 4 [](/posts/automate-90-of-your-design-job-search-with-this-ai-workflow-ijeybjd2c "Automate 90% of Your Design Job Search with This AI Workflow") Article UX Planet · 2y Automate 90% of Your Design Job Search with This AI Workflow Discover an AI-driven workflow to automate your job search process. This system uses tools like Bardeen for scraping job listings and Google Sheets with App Scripts to filter and organize them. It then leverages GPT-40 mini to analyze job descriptions based on specific criteria, helping you focus on the most relevant opportunities quickly. 161 4 5. 5 [](/posts/building-a-netflix-show-recommender-using-crawlee-and-react-px29flr2v "Building a Netflix show recommender using Crawlee and React") Article Community Picks · 2y Building a Netflix show recommender using Crawlee and React Learn how to build a Netflix show recommender using Crawlee and React, guided through scraping Netflix content with Crawlee and visualizing it with a React app built with Vite. The guide covers prerequisites, installation steps, writing scraping code using Cheerio, and integrating the scraped data into a React application. 153 6. 6 [](/posts/this-is-how-i-scrape-99-websites-via-llm-6ldxunnn3 "This is how I scrape 99% websites via LLM") Video YouTube · 2y This is how I scrape 99% websites via LLM Explore how advancements in AI, particularly large language models (LLMs), are revolutionizing web scraping in 2024\. Learn the best practices for scripting internet data at a large scale, building autonomous web scrapers, and handling complex web interactions. The post demonstrates various kinds of web scraping tasks, including scraping public websites, handling complex web manipulations, and more sophisticated tasks that require reasoning. It also includes details about services like OpenAI, AgentQL, and SpiderCloud that facilitate optimized web content extraction. 152 7. 7 [](/posts/python-ai-web-scraper-tutorial---use-ai-to-scrape-anything-qvtnlkrx6 "Python AI Web Scraper Tutorial - Use AI To Scrape ANYTHING") Video Tech With Tim · 2y Python AI Web Scraper Tutorial - Use AI To Scrape ANYTHING This post guides you on building an AI-powered web scraper using Python. The scraper can extract information from any website by passing a URL and a prompt to the AI. Essential tools include Streamlit for the front end, Selenium for web scraping, and Langchain for integrating with AI models. Detailed steps cover setting up the environment, handling dependencies, and developing the UI and backend functions necessary for scraping and parsing web content. The tuto­r­ial also explores overcoming common challenges like captchas and IP bans using Bright Data's scraping browser. 119 3 8. 8 [](/posts/how-to-scrape-google-maps-data-using-python-hq9ewxjau "How to scrape Google Maps data using Python") Article Community Picks · 2y How to scrape Google Maps data using Python Learn how to build a Google Maps scraper using Crawlee and Python to extract hotel data including names, ratings, reviews, prices, and amenities. The guide covers setting up the environment, connecting to Google Maps, handling dynamic content, and managing infinite scrolling. It also explains how to use proxies for large-scale scraping and create an interactive analysis dashboard with the exported data. 90 4 9. 9 [](/posts/tracking-supermarket-prices-with-playwright-tn70ffdxy "Tracking supermarket prices with playwright") Article Hacker News · 2y Tracking supermarket prices with playwright In Dec 2022, a website was created to track price changes in Greece's largest supermarkets using Playwright for web scraping. The main challenges included handling JavaScript-based sites, automating the scraping process, and avoiding IP restrictions. After initial attempts with an old laptop failed, a decision was made to use Hetzner for its cost-efficiency. The setup integrated Tailscale to tackle IP restrictions and used a CI server to manage daily scraping tasks. Optimizations focused on improving scrape speed and cost-efficiency, like upgrading server specs and reducing data fetched. 89 2 10. 10 [](/posts/i-built-my-first-saas---notifast-awsgdgjjo "I built my first SaaS - NotiFast") Article DEV · 2y I built my first SaaS - NotiFast NotiFast is a versatile notification bot designed to alert users about changes on websites they follow, such as new items or content updates. Built on the Discord platform, it offers seamless notifications without requiring user authentication and integrates easily with Discord's payment system. Initially derived from the open-source project webscraper-bot, NotiFast aims to simplify webpage monitoring with an easy-to-use visual creator. A free beta is currently available for the first 100 users. 87 2 11. 11 [](/posts/coding-project-ideas-for-portfolio-u2ek5okim "Coding project IDEAS for portfolio") Video Community Picks · 3y Coding project IDEAS for portfolio Learn how to find real-life coding projects for your portfolio. Get inspiration for web design, web scraping, data analytics, Chrome extensions, and full-stack applications. 85 4 12. 12 [](/posts/raznem-parsera-lightweight-library-for-scraping-web-sites-with-llms-8ppg0bxok "raznem/parsera: Lightweight library for scraping web-sites with LLMs") Article Community Picks · 2y raznem/parsera: Lightweight library for scraping web-sites with LLMs Parsera is a lightweight Python library designed for scraping websites using large language models (LLMs). It is easy to set up with minimal token use, boosting speed and reducing costs. Users can configure it to use models from OpenAI or Azure, and it includes asynchronous support. The library can extract specified elements from web pages and return the results in JSON format. 84 13. 13 [](/posts/vercel-puppeteer-mgsrinmb2 "Vercel + Puppeteer") Article Community Picks · 2y Vercel + Puppeteer Learn how to use Puppeteer with Vercel to generate PDFs of websites. Discover best practices for setting up Puppeteer for Vercel and deploying your Puppeteer code on Vercel. 84 5 14. 14 [](/posts/firecrawl-a-powerful-web-scraping-tool-for-turning-websites-into-large-language-model-llm-ready-m-xfv64it5y "Firecrawl: A Powerful Web Scraping Tool for Turning Websites into Large Language Model (LLM) Ready Markdown or Structured Data") Article Machine Learning News · 2y Firecrawl: A Powerful Web Scraping Tool for Turning Websites into Large Language Model (LLM) Ready Markdown or Structured Data Firecrawl, developed by Mendable AI, is a state-of-the-art web scraping tool designed to efficiently extract data from websites, including those with dynamic JavaScript-rendered content. It outputs clean, well-formatted Markdown suitable for Large Language Model (LLM) applications, while incorporating caching mechanisms and generative feedback loops to enhance data quality and extraction efficiency. Users can access Firecrawl via an intuitive API and multiple SDKs for different programming languages. 67 3 15. 15 [](/posts/link-eater-the-whatsapp-bot-that-digests-content-for-you-aq0qfb53i "Link Eater: The WhatsApp Bot That Digests Content for You") Article DEV · 2y Link Eater: The WhatsApp Bot That Digests Content for You Link Eater is an AI-powered bot that summarizes web content and YouTube videos directly through WhatsApp, allowing users to quickly grasp key points without navigating through lengthy content. It uses technologies like Node.js, Express.js, Twilio WhatsApp API, OpenAI GPT-3.5, YouTube Data API, and Jina AI. Users can send a URL to the bot, which then responds with a concise summary, making it useful for quick information digestion while engaging in chats and discussions. 65 3 16. 16 [](/posts/how-to-use-the-python-sdk-to-build-your-own-web-scraper-v19p3fdfu "How to Use the Python SDK to Build Your Own Web Scraper") Article freeCodeCamp · 2y How to Use the Python SDK to Build Your Own Web Scraper Learn how to use Python's Requests and Beautiful Soup libraries to build your own web scraper. This guide walks through scraping data from the UC Irvine Machine Learning Repository, covering the necessary libraries, defining functions to scrape and parse data, and saving the data to a CSV file. Important considerations include legal guidelines, ethical practices, and website compliance. 64 4 17. 17 [](/posts/one-million-screenshots-dbqy9lfm3 "One Million Screenshots") Article Community Picks · 2y One Million Screenshots Explore over a million rendered homepages from the web in an interactive manner, allowing you to zoom, pan, and click similar to Google Maps. This visual dataset could help you find websites you've been looking for or discover new ones. Check out the FAQ for more details and learn about the Screenshot API if you're interested in the data. 57 10 18. 18 [](/posts/7-python-libraries-every-data-engineer-should-know-3klrdr7pf "7 Python Libraries Every Data Engineer Should Know") Article KDnuggets · 2y 7 Python Libraries Every Data Engineer Should Know Discover some essential Python libraries for data engineers, including Requests for API data extraction, BeautifulSoup for web scraping, Pandas for data manipulation, SQLAlchemy for database work, Airflow for workflow orchestration, PySpark for big data processing, and Kafka-Python for real-time data processing. 52 19. 19 [](/posts/meet-steel-dev-an-open-source-browser-api-for-ai-agents-and-apps-y8beyay5v "Meet Steel.dev: An Open Source Browser API for AI Agents and Apps") Article Machine Learning News · 2y Meet Steel.dev: An Open Source Browser API for AI Agents and Apps Steel.dev is an open-source tool that simplifies web automation for AI applications by abstracting complex browser interactions through a RESTful API. It reduces the need for detailed scripts and expertise in frameworks like Puppeteer, Selenium, and Playwright. The tool features a modular architecture that allows easy management and interaction with headless browsers, facilitating tasks such as data extraction and form completion while ensuring scalability for large-scale projects. 51 20. 20 [](/posts/web-scraping-in-python---the-complete-guide-y4mq2oy3z "Web Scraping in Python - The Complete Guide") Article Hacker News · 3y Web Scraping in Python - The Complete Guide This post provides a comprehensive guide to web scraping in Python, including the advantages of using Python for web scraping, the best Python libraries for web scraping, and some tips and best practices for handling challenges in web scraping. It also includes code examples and recommendations for alternative libraries and tools for web scraping. 51 2 21. 21 [](/posts/finic-ai-finic-create-playwright-based-browser-agents-to-scrape-websites-and-automate-tasks--p72qhmbpz "finic-ai/finic: Create Playwright-based browser agents to scrape websites and automate tasks.") Article Hacker News · 2y finic-ai/finic: Create Playwright-based browser agents to scrape websites and automate tasks. Finic is a cloud platform that simplifies deploying and managing browser-based automation agents, focusing on fault-tolerant execution. It supports Playwright for DOM interaction and BeautifulSoup for HTML parsing. Features include cloud deployment, secure credential management, monitoring, and advanced functionalities like self-healing selectors and session impersonation. 49 1 22. 22 [](/posts/introduction-to-web-scraping-with-python-real-python-m2nw4twtv "Introduction to Web Scraping With Python – Real Python") Article Real Python · 2y Introduction to Web Scraping With Python – Real Python Web scraping is the process of collecting and parsing raw data from the web using powerful Python tools. This video course offers 12 lessons covering methods such as string methods, regular expressions, and HTML parsing. It includes downloadable resources, subtitles, transcripts, an interactive quiz, and a certificate of completion to help you effectively scrape data from websites. 47 23. 23 [](/posts/web-scraping-as-an-api-service-8wvu5cpus "Web scraping as an API service") Article Planet Python · 2y Web scraping as an API service This post discusses the use of web scraping as an API service in systems-to-systems integrations. It highlights why web scraping should be avoided in backend integrations and introduces Playwright as a tool for generating Python code for web scraping. 45 24. 24 [](/posts/scrapegraphai-a-web-scraping-python-library-that-uses-llms-to-create-scraping-pipelines-for-website-odweaorrw "ScrapeGraphAI: A Web Scraping Python Library that Uses LLMs to Create Scraping Pipelines for Websites, Documents, and XML Files") Article Machine Learning News · 2y ScrapeGraphAI: A Web Scraping Python Library that Uses LLMs to Create Scraping Pipelines for Websites, Documents, and XML Files ScrapeGraphAI is an advanced web scraping library that simplifies data collection using large language models (LLMs) and a unique direct graph logic. It minimizes the time and technical skills required for web scraping projects, allowing users to focus more on analyzing the extracted data. 45 3 25. 25 [](/posts/flaresolverr-flaresolverr-proxy-server-to-bypass-cloudflare-protection-hjwlqqrxz "FlareSolverr/FlareSolverr: Proxy server to bypass Cloudflare protection") Article Community Picks · 2y FlareSolverr/FlareSolverr: Proxy server to bypass Cloudflare protection FlareSolverr is a proxy server designed to bypass Cloudflare and DDoS-GUARD protection using Selenium with an undetected Chrome driver. It opens URLs with user parameters, solves Cloudflare challenges, and returns HTML code and cookies. Installation using Docker is recommended due to its dependencies on an external browser. Memory consumption can be high, so it should be used cautiously on low-RAM machines. It supports multiple architectures and provides examples for making requests via Bash, Python, and PowerShell. Users can also create permanent sessions to avoid repeatedly solving challenges. 43 1 [See all Crawling archives](/tags/crawling/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/tags/crawling/best-of/2024#page","url":"https://daily.dev/tags/crawling/best-of/2024","name":"Best Crawling Posts of 2024","description":"The most upvoted Crawling posts of 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/tags/crawling/best-of/2024#items","numberOfItems":25,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/using-gpt-4o-for-web-scraping-orrhtenf9","name":"Using GPT-4o for web scraping"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/7-open-source-projects-you-should-know---python-edition--ot12y3hmc","name":"7 Open Source Projects You Should Know - Python Edition ✔️"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/the-best-project-idea-to-learn-aws-yvitbmafy","name":"The BEST Project Idea to Learn AWS"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/automate-90-of-your-design-job-search-with-this-ai-workflow-ijeybjd2c","name":"Automate 90% of Your Design Job Search with This AI Workflow"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/building-a-netflix-show-recommender-using-crawlee-and-react-px29flr2v","name":"Building a Netflix show recommender using Crawlee and React"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/this-is-how-i-scrape-99-websites-via-llm-6ldxunnn3","name":"This is how I scrape 99% websites via LLM"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/python-ai-web-scraper-tutorial---use-ai-to-scrape-anything-qvtnlkrx6","name":"Python AI Web Scraper Tutorial - Use AI To Scrape ANYTHING"},{"@type":"ListItem","position":8,"url":"https://daily.dev/posts/how-to-scrape-google-maps-data-using-python-hq9ewxjau","name":"How to scrape Google Maps data using Python"},{"@type":"ListItem","position":9,"url":"https://daily.dev/posts/tracking-supermarket-prices-with-playwright-tn70ffdxy","name":"Tracking supermarket prices with playwright"},{"@type":"ListItem","position":10,"url":"https://daily.dev/posts/i-built-my-first-saas---notifast-awsgdgjjo","name":"I built my first SaaS - NotiFast"},{"@type":"ListItem","position":11,"url":"https://daily.dev/posts/coding-project-ideas-for-portfolio-u2ek5okim","name":"Coding project IDEAS for portfolio"},{"@type":"ListItem","position":12,"url":"https://daily.dev/posts/raznem-parsera-lightweight-library-for-scraping-web-sites-with-llms-8ppg0bxok","name":"raznem/parsera: Lightweight library for scraping web-sites with LLMs"},{"@type":"ListItem","position":13,"url":"https://daily.dev/posts/vercel-puppeteer-mgsrinmb2","name":"Vercel + Puppeteer"},{"@type":"ListItem","position":14,"url":"https://daily.dev/posts/firecrawl-a-powerful-web-scraping-tool-for-turning-websites-into-large-language-model-llm-ready-m-xfv64it5y","name":"Firecrawl: A Powerful Web Scraping Tool for Turning Websites into Large Language Model (LLM) Ready Markdown or Structured Data"},{"@type":"ListItem","position":15,"url":"https://daily.dev/posts/link-eater-the-whatsapp-bot-that-digests-content-for-you-aq0qfb53i","name":"Link Eater: The WhatsApp Bot That Digests Content for You"},{"@type":"ListItem","position":16,"url":"https://daily.dev/posts/how-to-use-the-python-sdk-to-build-your-own-web-scraper-v19p3fdfu","name":"How to Use the Python SDK to Build Your Own Web Scraper"},{"@type":"ListItem","position":17,"url":"https://daily.dev/posts/one-million-screenshots-dbqy9lfm3","name":"One Million Screenshots"},{"@type":"ListItem","position":18,"url":"https://daily.dev/posts/7-python-libraries-every-data-engineer-should-know-3klrdr7pf","name":"7 Python Libraries Every Data Engineer Should Know"},{"@type":"ListItem","position":19,"url":"https://daily.dev/posts/meet-steel-dev-an-open-source-browser-api-for-ai-agents-and-apps-y8beyay5v","name":"Meet Steel.dev: An Open Source Browser API for AI Agents and Apps"},{"@type":"ListItem","position":20,"url":"https://daily.dev/posts/web-scraping-in-python---the-complete-guide-y4mq2oy3z","name":"Web Scraping in Python - The Complete Guide"},{"@type":"ListItem","position":21,"url":"https://daily.dev/posts/finic-ai-finic-create-playwright-based-browser-agents-to-scrape-websites-and-automate-tasks--p72qhmbpz","name":"finic-ai/finic: Create Playwright-based browser agents to scrape websites and automate tasks."},{"@type":"ListItem","position":22,"url":"https://daily.dev/posts/introduction-to-web-scraping-with-python-real-python-m2nw4twtv","name":"Introduction to Web Scraping With Python – Real Python"},{"@type":"ListItem","position":23,"url":"https://daily.dev/posts/web-scraping-as-an-api-service-8wvu5cpus","name":"Web scraping as an API service"},{"@type":"ListItem","position":24,"url":"https://daily.dev/posts/scrapegraphai-a-web-scraping-python-library-that-uses-llms-to-create-scraping-pipelines-for-website-odweaorrw","name":"ScrapeGraphAI: A Web Scraping Python Library that Uses LLMs to Create Scraping Pipelines for Websites, Documents, and XML Files"},{"@type":"ListItem","position":25,"url":"https://daily.dev/posts/flaresolverr-flaresolverr-proxy-server-to-bypass-cloudflare-protection-hjwlqqrxz","name":"FlareSolverr/FlareSolverr: Proxy server to bypass Cloudflare protection"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tags","item":"https://daily.dev/tags"},{"@type":"ListItem","position":3,"name":"Crawling","item":"https://daily.dev/tags/crawling"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/tags/crawling/best-of"},{"@type":"ListItem","position":5,"name":"2024"}]}]} ```