A mathematical walkthrough derives linear regression and Poisson regression as special cases of the Expectation-Maximization (EM) algorithm. For linear regression, the E-step collapses trivially and the M-step reproduces the closed-form OLS solution in a single iteration. For Poisson regression, a second-order Taylor expansion yields a weighted least-squares Q-function, and the M-step becomes the Iteratively Reweighted Least Squares (IRLS) update used in generalized linear models, requiring multiple iterations to converge since the closed-form solution doesn't exist. References to Casella and Berger, and McCullagh and Nelder are given for further reading.
Questions this post answers
Why does the EM algorithm converge in a single iteration for linear regression?
The M-step yields the global maximum of the Q-function in closed form, and that maximum does not depend on the current parameter estimate. Because ordinary least squares already has a closed-form solution β=(XᵀX)⁻¹Xᵀy, plugging in any starting value produces the same result after one M-step, since the Q-function's maximizer is independent of β(t). daily.dev surfaces deep dives like this for developers building statistical or ML models with regression.
How does the EM algorithm derive the IRLS update used for Poisson regression?
A second-order Taylor expansion of the exponential mean function around the current linear predictor turns the Poisson log-likelihood into a weighted least-squares Q-function, with weights equal to the current fitted means μ(t) and working response ỹ(t)=η(t)+(yi−μ(t))/μ(t). Maximizing this gives β(t+1)=(XᵀW(t)X)⁻¹XᵀW(t)ỹ(t), the standard Iteratively Reweighted Least Squares update for GLMs. engineers implementing GLM solvers can find similar derivations curated on daily.dev.