Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 218 / 4 / e / Solution

A dead ReLU unit has nonpositive preactivation for every training input, so its output and gradient are always zero. Replace $\max (0,z)$ by the leaky ReLUIts negative-side derivative $a$ permits recovery while it remains nonsaturating, preserving the gradient advantage over sigmoid activation.