JavaScript in Plain English
Read post

Why Increasing Concurrency Causes HTTP 429 Errors (It’s Not Just Too Many Requests)

Increasing concurrency in HTTP clients doesn't just raise the number of in-flight requests — it also creates a burst of simultaneous request starts at t=0 and unpredictable per-host concurrency in multi-host workloads. Both hidden factors can trigger HTTP 429 rate-limit errors even when your global pool size seems reasonable. Benchmarks show that adding a start-rate cap (independent of pool size) can recover useful throughput from 16/100 to 100/100 successful requests at the same concurrency level. For multi-host workloads, a per-host limiter nested inside the global pool prevents any single strict host from absorbing a disproportionate burst. Residential proxies are also discussed as a workaround that raises the effective burst tolerance without fixing the root cause. A practical checklist covers measuring observed start RPS, per-host peak in-flight counts, and optimizing for successful requests rather than raw completed request rate.

    #javascript#crawling
Jul 24•18m read time•From javascript.plainenglish.io
Post cover image
Table of contents
This guide shows how start-rate bursts, per-host concurrency, and rate limits — not just worker count — determine real throughput.GitHub - sixthextinction/concurrency-trap-benchPart I — Concurrency Vs. Start Rate: Why One Pool Setting Controls Two Different LimitsMeasuring Start-Rate BurstUseful Throughput Falls As Global Concurrency Rises
470 Impressions
JavaScript in Plain English's image
JavaScript in Plain English

JSPlainEnglish's platform is a resource for JavaScript developers, offering insights into JavaScript...

99 Followers

•

799 Upvotes

Would you recommend this post?

Copy link
WhatsApp
Facebook
X
New Squad
  • © 2026 Daily Dev Ltd.
  • Guidelines
  • Explore
  • Tags
  • Sources
  • Squads
  • Leaderboard