A comprehensive tour of tree-based machine learning models for tabular data, covering the shared mechanics of all tree models and then walking through each major algorithm: Decision Trees (CART), Random Forest, Extra Trees, AdaBoost, Gradient Boosting, Histogram-based Gradient Boosting, XGBoost, LightGBM, and CatBoost. For each model, the post explains how it works, provides a minimal sklearn/library code snippet, and highlights the one property that distinguishes it from its neighbors. Key themes include the bagging vs. boosting taxonomy, bias-variance tradeoffs, and practical guidance on when to reach for each algorithm. The recommendation is to start with HistGradientBoostingClassifier for most tabular problems.
Table of contents
What every tree has in commonThe base unit: a single decision treeTwo ways to combine treesBagging: grow deep trees in parallel and average themBoosting: grow shallow trees in sequence and add them upTrees for other jobsPicking one179 Impressions