Past exam of the computer science course of the University of Oxford / 2015 / Computational learning theory / 3 / c

Write $\epsilon ={\epsilon }_t$, $\alpha ={\alpha }_t$, and $s_i=y_i{h}_t(x_i)\in \{-1,1\}$. Correctly classified observations have $s_i=1$ and total $D_t$-weight $1-\epsilon$; incorrectly classified observations have $s_i=-1$ and total weight $\epsilon$. The normalizing constant in the AdaBoost update is thereforeFor $0<\epsilon <1$, the definition of $\alpha$ givesIf observation $i$ is correct, its multiplier in the normalized update isif it is incorrect, the multiplier isThe corresponding denominators are exactly $1+s_i(1-2\epsilon )$: they are $2(1-\epsilon )$ when $s_i=1$ and $2\epsilon$ when $s_i=-1$. ConsequentlyAs a check, the new total mass on correctly classified observations is $(1-\epsilon )/(2(1-\epsilon ))=1/2$, and the mass on incorrectly classified observations is $\epsilon /(2\epsilon )=1/2$. The update thus produces a probability distribution and assigns the two groups equal aggregate mass. When $\epsilon =1/2$, $\alpha =0$ and every individual weight is unchanged.

The displayed finite update assumes $0<{\epsilon }_t<1$. The usual weak learner guarantees ${\epsilon }_t<1/2$, but it can return a perfect classifier with ${\epsilon }_t=0$; then the logarithm is infinite and the normalized exponential formula is not defined. AdaBoost stops on that perfect hypothesis rather than executing this update. Similarly, the printed formula is not an ordinary finite update at ${\epsilon }_t=1$.