A dead ReLU unit has nonpositive preactivation for every training input, so its output and gradient are always zero. Replace $\max(0,z)$ by the leaky ReLU
$$
\phi_a(z)=\max(z,az),\qquad0<a<1.
$$
Its negative-side derivative $a$ permits recovery while it remains nonsaturating, preserving the gradient advantage over sigmoid activation.

Solution

Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 218 / 4 / e / Solution

 Solution

ID: past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2021/iii/paper-218/4/e/solution