An in-depth mathematical breakdown explains why Random Forest adds feature subsampling on top of bagging. Starting from the variance-of-a-sum formula, it shows that correlated trees create a hard variance floor (ρσ²) that no amount of additional trees can overcome, since only the second term of the variance formula shrinks with n. Feature subsampling reduces the pairwise correlation ρ between trees at the cost of making individual trees noisier (higher σ²), but because the floor term scales with ρ, this trade nets a lower overall ensemble variance. A controlled experiment with 400 trials confirms the theory: plain bagging plateaus at a variance floor of 1.34, while Random Forest, despite individually noisier trees, plateaus at 0.76, ending up at roughly 59% of bagging's variance by 120 trees. The piece closes by arguing that max_features is an underrated hyperparameter that directly controls this ρ/σ² tradeoff and should be tuned against validation error rather than left at its default.
Table of contents
Bias-Variance, a Fast RefresherThe Mathematical Core: Discussing the variance computationWhy ρ Exists, and How Random Forest Breaks ItThe Experiment — What We're Actually TestingThe Subtle Point: Worse Trees, Better ForestPractical Takeaway: max_features Isn't a DetailAppendixReferences:Questions this post answers
Why does Random Forest randomly select a subset of features at each split instead of considering all features like bagging does?
Feature subsampling exists to break the correlation between trees, not to improve any single tree. When all trees can see every feature, they tend to split on the same dominant predictors, producing correlated errors that averaging cannot eliminate. By hiding the strongest features from some splits, Random Forest forces trees toward different structures, lowering the pairwise correlation ρ and, with it, the ensemble's variance floor. daily.dev surfaces deep dives like this for engineers deciding how to tune ensemble models.
Why can't averaging more trees in bagging keep reducing ensemble variance indefinitely?
Ensemble variance decomposes into two terms: one that shrinks toward zero as trees are added, and one, ρσ², that never disappears because it depends only on the average pairwise correlation ρ between trees, not on the tree count. In a synthetic experiment, plain bagging's variance plateaued at 1.34 no matter how many trees were added, while Random Forest's lower correlation dropped that floor to 0.76. Understanding this ceiling helps developers choosing between bagging and random forest for a given dataset.
Is a single Random Forest tree better or worse than a single bagged tree?
A single Random Forest tree is measurably worse than a single bagged tree because restricting each split to a random subset of features sometimes forces it away from the best available split. In one experiment, individual tree variance was 17.42 for Random Forest versus 9.89 for bagging, yet the full Random Forest ensemble still ended up with lower overall variance (0.85 vs 1.44 at 120 trees) because its trees were less correlated. daily.dev helps practitioners weighing individual model quality against ensemble-level tradeoffs.