Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 218 / 4 / c

Deep sigmoid networks suffer from vanishing gradients because derivatives become small in saturated units and multiply across layers. The ReLU has derivative one on its positive half-line and therefore propagates gradients more effectively there.