= Solution
Compose <affine maps> with nonlinear hidden <activation functions>, for example $h(x)=W_2\operatorname{ReLU}(W_1x+b_1)+b_2$. Several hidden units can respond to different linear combinations of input features; later units combine their responses to represent curved or piecewise-linear decision boundaries and interactions. A product interaction can be understood from $xy=[(x+y)^2-(x-y)^2]/4$: units approximating nonlinear functions of $x+y$ and $x-y$ can represent effects that no single additive linear fit captures.
\b[Nonlinearity in hidden layers is essential:] with only identity activations, repeated <affine maps> collapse to one <affine map> regardless of depth. Width and depth determine available representations; <regularization>, validation and suitable optimization are needed to learn useful rather than overfitted interactions. A nonlinear output activation alone applied to one linear score does not give an unrestricted model of <feature interactions>.
Back to article page
