= Solution
\b[The two limiting-case names in the printed bullets are reversed.] Fully reparameterized continuous choices give a <reparameterization gradient>; non-reparameterized discrete choices give a <score-function gradient estimator>. Here are both reductions, including the extra term that vanishes only after taking an <expectation>.
When all choices are reparameterized with parameter-independent base noise, $s_\lambda=s$ and $\nabla_\lambda\log s=0$. The expression in part (c) becomes
$$
\nabla_\lambda\mathcal L=\mathbb E_s\left[\nabla_\lambda\{\log p(t_\lambda(\varepsilon),x)-\log q_\lambda(t_\lambda(\varepsilon))\}\right].
$$
Differentiating along the sampled transformation is precisely the <pathwise gradient>, also called the <reparameterization gradient>. There is no <score function> of the parameter-independent base law.
When all choices are discrete and left untransformed, $t_\lambda(\varepsilon)=\varepsilon=z$ and $s_\lambda=q_\lambda$. Set $S_\lambda(z)=\nabla_\lambda\log q_\lambda(z)$ and $W_\lambda(z,x)=\log p(z,x)-\log q_\lambda(z)$. Since $p$ has no variational parameter, the integrand in part (c) is
$$
S_\lambda(z)W_\lambda(z,x)-S_\lambda(z)=S_\lambda(z)(W_\lambda(z,x)-1).
$$
<Normalization> gives the zero-mean <score function> identity
$$
\mathbb E_{q_\lambda}S_\lambda=\sum_z\nabla_\lambda q_\lambda(z)=\nabla_\lambda1=0.
$$
Thus
$$
\boxed{\nabla_\lambda\mathcal L=\mathbb E_{q_\lambda}[S_\lambda W_\lambda]=\mathbb E_{q_\lambda}[S_\lambda(W_\lambda-1)].}
$$
The first expression is the usual <score-function gradient estimator>; the second has a <constant score baseline> of one. The two random integrands are generally different even though their <expectations> agree. A suitable <constant score baseline> can reduce <variance>, but one is not universally the optimal baseline.<Gradient Estimation Using Stochastic Computation Graphs by John Schulman and others>[Section 4, Theorem 2 and footnote 3]{ref}
For a concrete check, take $q_\lambda(D,Z)=\operatorname{Bernoulli}(q)\,N(m,\sigma^2)$ with $q=(1+e^{-a})^{-1}$ and $\sigma=e^\ell$, and a fixed target $p(D,Z)=\operatorname{Bernoulli}(0.4)\,N(D,1)$. Direct integration gives
$$
\mathcal L=-D_{\rm KL}(\operatorname{Bern}(q)\Vert\operatorname{Bern}(0.4))-\tfrac12\{\sigma^2+m^2-2mq+q-1-2\ell\}.
$$
Using $Z=m+\sigma\varepsilon$, $\varepsilon\sim N(0,1)$, the combined integrand has components $(D-q)(W-1)$, $D-Z$ and $1-(Z-D)\sigma\varepsilon$. Their <expectations> are respectively $q(1-q)[\log(0.4/0.6)-\log(q/(1-q))+m-1/2]$, $q-m$ and $1-\sigma^2$, matching direct differentiation. Dropping the discrete <score function> term would incorrectly give zero for the first component. This calculation checks the combination without accepting the printed reversal.
Back to article page
