Solution
= Solution
The proposed function $g(\eta)=\eta\Phi(\eta)$ is the <Gaussian error linear unit>. Unlike ReLU, it is smooth at zero and has a nonzero gradient on much of the negative half-line, reducing dead hidden units and making gradient optimization smoother. It is less computationally convenient because evaluating the <standard normal cumulative distribution function> is costlier than taking a maximum, and it does not produce ReLU's exact sparse zero activations.