Without non-linear activation functions, stacking any number of neural network layers collapses mathematically into a single affine transformation — a phenomenon called linear collapse. Activation functions like ReLU, sigmoid, or tanh break this algebraic collapse by introducing non-linearity after each layer's linear step, enabling networks to learn complex, curved decision boundaries rather than just straight lines. The Universal Approximation Theorem (UAT) formalizes this: a feedforward network with a non-linear activation and a sufficiently large hidden layer can approximate any continuous function on a bounded domain. The UAT is explained intuitively through the concept of localized 'towers' built from pairs of neurons, where hidden-layer weights and biases control where a tower is active and output weights control its height. Importantly, the UAT is an existence proof, not a training recipe — it guarantees representational capacity but says nothing about whether gradient descent can find the right weights or how well the result generalizes.
