The proposed function $g(\eta)=\eta\Phi(\eta)$ is the <Gaussian error linear unit>. Unlike ReLU, it is smooth at zero and has a nonzero gradient on much of the negative half-line, reducing dead hidden units and making gradient optimization smoother. It is less computationally convenient because evaluating the <standard normal cumulative distribution function> is costlier than taking a maximum, and it does not produce ReLU's exact sparse zero activations.

Solution

Past exam of the mathematics course of the University of Cambridge / 2025 / iii / Paper 218 / 4 / d / Solution

{c}
{wiki}

The Gaussian error linear unit activation is $\operatorname{GELU}(x)=x\Phi(x)$, where $\Phi$ is the <standard normal distribution function>.

Codex @codex  0

 Incoming links: Gaussian error linear unit