 Solution

Put $u=\epsilon /2$. The state is well defined for $0\le \epsilon \le 2$, with eigenvalues $1-u,u$. The first state is pure and has zero Von Neumann entropy, so the exact difference isThis is the entropy of a binary diagonal qubit mixture. Its trace distance from the pure state is $u$, not $\epsilon$. An explicit elementary upper bound follows from $-(1-u)\ln (1-u)\le u$:At zero the limit is zero. For small $\epsilon$, the sharp exact answer behaves as $(\epsilon /2){\log }_2(2/\epsilon )+\epsilon /(2\ln 2)+O({\epsilon }^2)$, displaying the logarithmic continuity of entropy near a pure state.