Deep sigmoid networks suffer from vanishing gradients because derivatives become small in saturated units and multiply across layers. The <ReLU> has derivative one on its positive half-line and therefore propagates gradients more effectively there.

Solution

Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 218 / 4 / c / Solution

 Solution (source code)