Past exam of the mathematics course of the University of Cambridge / 2025 / iii / Paper 218 / 4 / d

The proposed function $g(\eta )=\eta \Phi (\eta )$ is the Gaussian error linear unit. Unlike ReLU, it is smooth at zero and has a nonzero gradient on much of the negative half-line, reducing dead hidden units and making gradient optimization smoother. It is less computationally convenient because evaluating the standard normal cumulative distribution function is costlier than taking a maximum, and it does not produce ReLU's exact sparse zero activations.