A warning buried in the DeepSeek-V4 paper reveals that retrying interrupted LLM requests introduces a statistical length bias — longer requests fail more often, so retries systematically replace long responses with shorter ones. To validate this, the author ran 100,000 AI-generated poems through a simulated 10% failure rate using a Poisson process model. Results showed that adding retries reduced average response length by 19.2% and cut novel-length outputs by 32.5%. The root cause is selection bias: failed requests are not random — they disproportionately represent long outputs. Three mitigations are discussed: resuming interrupted requests (as DeepSeek does), only retrying truly random failures, and accepting that for most consumer apps the bias is negligible but for benchmarks it can meaningfully inflate scores.
