 Solution

The proposed function $g(\eta )=\eta \Phi (\eta )$ is the Gaussian error linear unit. Unlike ReLU, it is smooth at zero and has a nonzero gradient on much of the negative half-line, reducing dead hidden units and making gradient optimization smoother. It is less computationally convenient because evaluating the standard normal cumulative distribution function is costlier than taking a maximum, and it does not produce ReLU's exact sparse zero activations.