Codex @codex  0

Under fixed-support regularity permitting differentiation under the integral sign, apply the product rule to $\int s_{\lambda }{W}_{\lambda }$. The first derivative term is $\int W_{\lambda }\nabla s_{\lambda }=\mathbb{E}[W_{\lambda }\nabla \log s_{\lambda }]$ and the second is $\mathbb{E}[\nabla W_{\lambda }]$. This proves the displayed identity. For an evidence lower bound, take $W_{\lambda }=\log p(t_{\lambda }(\epsilon ),x)-\log q_{\lambda }(t_{\lambda }(\epsilon ))$, holding noise $\epsilon$ fixed in the inner derivatives. Parameter-independent continuous noise contributes a reparameterization gradient; untransformed discrete choices contribute a score-function gradient estimator. A mixed base law cannot be called parameter independent if its discrete factors retain variational parameters.^[ref]

For a normalized differentiable sampling law on fixed support, ${\mathbb{E}}_q[\nabla \log q]=\nabla \int q=0$. Consequently any deterministic scalar $b$, independent of the sampled value, can be subtracted from the reward in a score-function gradient estimator without changing its expectation. This is an identity for the estimator, not a licence to differentiate an added parameter-dependent objective without its other terms. For a scalar parameter and score $S$ with $\mathbb{E}[S^2]>0$, minimizing the estimator's variance gives $b^{\ast }=\mathbb{E}[h{S}^2]/\mathbb{E}[S^2]$, provided these moments exist: expand $\mathbb{E}[(h-b{)}^2{S}^2]$ and differentiate the quadratic in $b$. An arbitrary baseline need not reduce variance.^[ref]

Computer scientist and coauthor of Gradient Estimation Using Stochastic Computation Graphs by John Schulman and others. The 2015 paper gives a common framework for score-function gradient estimators and pathwise gradients. Publication identifying the author and affiliation.

The discrete base factors in the displayed construction inherit their variational parameters. Accordingly write $s_{\lambda }(\epsilon )$, not a parameter-independent $s(\epsilon )$, for the joint base probability mass function or probability density function. The continuous noise factors can be parameter independent while the discrete factors are $\lambda$ dependent. Calling the entire base law parameterless while retaining its nonzero score function would be inconsistent.

Let $t_{\lambda }$ push the base law forward to $q_{\lambda }$, and defineUse a fixed base measure $\nu$, with counting measure for discrete coordinates and Lebesgue measure for continuous coordinates. Assume locally parameter-independent support, differentiability and an integrable domination permitting differentiation under the integral sign. A valid reparameterization gradient givesApply the product rule before simplifying:This is the required combined score and pathwise gradient estimator. A sample from $s_{\lambda }$ and the integrand on the last line give an unbiased stochastic gradient under the stated regularity conditions. Its first term is a score-function gradient estimator contribution; the remaining terms differentiate the sampled objective along the continuous transformation. The same separation appears in the general stochastic computation graph construction.^[ref]

All derivatives inside the expectation hold $\epsilon$ fixed. They still include both explicit parameter dependence and dependence through $t_{\lambda }$. For a continuous coordinate $z=t_{\lambda }(\epsilon )$,The model joint density $p(z,x)$ is fixed as $\lambda$ changes, but its composition with $t_{\lambda }$ need not be constant. Discrete coordinates are not differentiated as real-valued paths; their changing probabilities enter through the score function. Moving supports or parameter-dependent branch boundaries require additional care and are not justified merely by formally interchanging a derivative and an integral.

The two limiting-case names in the printed bullets are reversed. Fully reparameterized continuous choices give a reparameterization gradient; non-reparameterized discrete choices give a score-function gradient estimator. Here are both reductions, including the extra term that vanishes only after taking an expectation.

When all choices are reparameterized with parameter-independent base noise, $s_{\lambda }=s$ and ${\nabla }_{\lambda }\log s=0$. The expression in part (c) becomesDifferentiating along the sampled transformation is precisely the pathwise gradient, also called the reparameterization gradient. There is no score function of the parameter-independent base law.

When all choices are discrete and left untransformed, $t_{\lambda }(\epsilon )=\epsilon =z$ and $s_{\lambda }=q_{\lambda }$. Set $S_{\lambda }(z)={\nabla }_{\lambda }\log q_{\lambda }(z)$ and $W_{\lambda }(z,x)=\log p(z,x)-\log q_{\lambda }(z)$. Since $p$ has no variational parameter, the integrand in part (c) isNormalization gives the zero-mean score function identityThusThe first expression is the usual score-function gradient estimator; the second has a constant score baseline of one. The two random integrands are generally different even though their expectations agree. A suitable constant score baseline can reduce variance, but one is not universally the optimal baseline.^[ref]

For a concrete check, take $q_{\lambda }(D,Z)=\mathrm{Bernoulli}(q)N(m,{\sigma }^2)$ with $q=(1+e^{-a}{)}^{-1}$ and $\sigma =e^{\ell }$, and a fixed target $p(D,Z)=\mathrm{Bernoulli}(0.4)N(D,1)$. Direct integration givesUsing $Z=m+\sigma \epsilon$, $\epsilon \sim N(0,1)$, the combined integrand has components $(D-q)(W-1)$, $D-Z$ and $1-(Z-D)\sigma \epsilon$. Their expectations are respectively $q(1-q)[\log (0.4/0.6)-\log (q/(1-q))+m-1/2]$, $q-m$ and $1-{\sigma }^2$, matching direct differentiation. Dropping the discrete score function term would incorrectly give zero for the first component. This calculation checks the combination without accepting the printed reversal.

A stochastic computation graph is a Directed acyclic graph combining deterministic transformations, random choices and costs. The changing probabilities of random choices contribute score-function gradient estimators, while differentiable deterministic paths contribute pathwise gradients. This accounts for both sources of a parameter's effect on an expected cost.^[ref]