A production engineering team built an AI-driven event aggregation scraper that ingests data from hundreds of partner sites. The post covers the full pipeline: handling static HTML vs SPAs vs lazy-loaded content using Playwright, using LLMs to generate CSS selectors (which fail 30-40% of the time on first attempt), reducing HTML size before sending to LLMs, preferring stable selectors like JSON-LD and data-testid over hashed class names, dealing with bot detection tiers, and implementing a human-in-the-loop correction loop with structured failure diagnosis. Key tools include Playwright, playwright-stealth, BeautifulSoup, Pydantic, and LlamaIndex. The core lesson: the LLM handles pattern recognition, while validation, retry logic, and failure diagnosis are engineering problems that wrap around it.
