--- title: XGBoost from Scratch | daily.dev description: A from-scratch walkthrough of how XGBoost builds a single decision tree, covering the math behind log-odds initial predictions, gradients and hessians, split... canonical: https://daily.dev/posts/xgboost-from-scratch-uw523v5zy twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: XGBoost from Scratch | daily.dev og:description: A from-scratch walkthrough of how XGBoost builds a single decision tree, covering the math behind log-odds initial predictions, gradients and hessians, split... og:url: https://daily.dev/posts/xgboost-from-scratch-uw523v5zy og:image: https://api.daily.dev/og/posts/Uw523V5zy.png og:image:alt: XGBoost from Scratch og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # XGBoost from Scratch **[Medium](https://daily.dev/sources/medium_js)** · 9 min read · 0 upvotes · 0 comments ## Summary A from-scratch walkthrough of how XGBoost builds a single decision tree, covering the math behind log-odds initial predictions, gradients and hessians, split gain calculation with L2 regularization (lambda), leaf penalties (gamma), leaf weight computation, and prediction updates via learning rate (eta). Uses a small 6-row toy dataset predicting job acceptance to manually work through each step, explaining how lambda disproportionately regularizes low-hessian (small) splits and how gamma can turn positive gains negative to stop tree growth. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does L2 regularization (lambda) affect XGBoost split gain calculations differently for small vs large splits? Lambda affects the divisor in the gain formula GL²/(HL+lambda), and its impact depends on the hessian size. When a split covers many examples, such as a hessian of 100, adding lambda of 1 barely changes the result. But for a split with only a few examples, such as a hessian of 0.25, adding lambda of 1 drastically alters the divisor, penalizing small, overfit-prone splits far more heavily than large ones. _daily.dev surfaces deeper explainers like this for engineers tuning regularization in gradient boosting models._ ### What do the gamma and eta parameters control in XGBoost? Gamma is the leaf penalty representing the minimum loss reduction required to make a further split, so raising it (for example from 0 to 0.5) can turn a positive gain negative and stop the tree from splitting further. Eta is the learning rate or shrinkage size that dilutes how much each new tree's weight updates affect the running prediction, spreading corrections across more iterations. _Developers tuning boosting hyperparameters can find grounded walkthroughs like this via daily.dev._ ### Why does XGBoost use log odds instead of probabilities for its raw prediction score? XGBoost needs continuous, unbounded values to apply corrections during boosting because probabilities are bounded between 0 and 1 and unbounded updates would produce nonsensical values like 1.3 or -0.2. Log odds is unbounded, so gradients and hessians are computed on it, and the result is passed through a sigmoid function to convert it back into a probability for interpretation. _daily.dev helps developers dig into the underlying mechanics behind popular boosting algorithms like XGBoost._ ## Similar posts on daily.dev - [My first experiment with gradient boosting](https://daily.dev/posts/my-first-experiment-with-gradient-boosting-tjnxetq3w) · Medium · 0 upvotes · 0 comments - [The Machine Learning “Advent Calendar” Day 21: Gradient Boosted Decision Tree Regressor in Excel](https://daily.dev/posts/the-machine-learning-advent-calendar-day-21-gradient-boosted-decision-tree-regressor-in-excel-7hgnbjjwp) · Towards Data Science · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#xgboost](https://daily.dev/tags/xgboost) [View this post on daily.dev](https://daily.dev/posts/xgboost-from-scratch-uw523v5zy) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"XGBoost from Scratch","url":"https://daily.dev/posts/xgboost-from-scratch-uw523v5zy","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/xgboost-from-scratch-uw523v5zy"},"datePublished":"2026-08-30T22:36:29.505Z","dateModified":"2026-08-30T22:49:23.975Z","description":"A from-scratch walkthrough of how XGBoost builds a single decision tree, covering the math behind log-odds initial predictions, gradients and hessians, split...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ee0ffdb54f7b49b010650c67ac6174ff?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ee0ffdb54f7b49b010650c67ac6174ff?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/xgboost-from-scratch-uw523v5zy","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,xgboost","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"XGBoost from Scratch"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/xgboost-from-scratch-uw523v5zy#faq","mainEntity":[{"@type":"Question","name":"How does L2 regularization (lambda) affect XGBoost split gain calculations differently for small vs large splits?","acceptedAnswer":{"@type":"Answer","text":"Lambda affects the divisor in the gain formula GL²/(HL+lambda), and its impact depends on the hessian size. When a split covers many examples, such as a hessian of 100, adding lambda of 1 barely changes the result. But for a split with only a few examples, such as a hessian of 0.25, adding lambda of 1 drastically alters the divisor, penalizing small, overfit-prone splits far more heavily than large ones. daily.dev surfaces deeper explainers like this for engineers tuning regularization in gradient boosting models."}},{"@type":"Question","name":"What do the gamma and eta parameters control in XGBoost?","acceptedAnswer":{"@type":"Answer","text":"Gamma is the leaf penalty representing the minimum loss reduction required to make a further split, so raising it (for example from 0 to 0.5) can turn a positive gain negative and stop the tree from splitting further. Eta is the learning rate or shrinkage size that dilutes how much each new tree's weight updates affect the running prediction, spreading corrections across more iterations. Developers tuning boosting hyperparameters can find grounded walkthroughs like this via daily.dev."}},{"@type":"Question","name":"Why does XGBoost use log odds instead of probabilities for its raw prediction score?","acceptedAnswer":{"@type":"Answer","text":"XGBoost needs continuous, unbounded values to apply corrections during boosting because probabilities are bounded between 0 and 1 and unbounded updates would produce nonsensical values like 1.3 or -0.2. Log odds is unbounded, so gradients and hessians are computed on it, and the result is passed through a sigmoid function to convert it back into a probability for interpretation. daily.dev helps developers dig into the underlying mechanics behind popular boosting algorithms like XGBoost."}}]} ```