100% Automated: How I Built a Self-Healing Price Scraper for 163 Shipping Providers
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A solo engineer describes building a fully automated, self-healing price scraper for 163 European shipping providers using an agentic task graph. The system runs updater scripts per provider, detects failures, and dispatches AI agents (Claude Code CLI, OpenAI Codex CLI, Cursor) to diagnose broken scrapers, rewrite scripts, and verify prices against live carrier websites. Key design patterns include: a layered architecture separating the calculator, updater scripts, and agentic loop; pluggable LLM routing with fallbacks and cooldown handling for rate limits; deterministic pre-checks before invoking LLMs; and a two-agent verifier pattern where a second agent reviews the first's output cold. Feedback from failed verifications is written into the next agent's prompt as structured context. The post concludes with practical takeaways for engineers building agentic pipelines, including single-responsibility agents, independent verifiers, and model-to-task matching.
