Score-function gradient estimator (${\nabla }_{\lambda }{\mathbb{E}}_{q_{\lambda }}h={\mathbb{E}}_{q_{\lambda }}[h{\nabla }_{\lambda }\log q_{\lambda }]$)

For a parameter-independent integrand $h$, the displayed score-function gradient estimator follows by differentiating the sampling probability density function or probability mass function, using $\nabla q=q\nabla \log q$. It requires conditions permitting differentiation under the integral sign, or the corresponding sum, and control of support changes. It applies to discrete choices without pretending that their sampled values are differentiable paths. If $h=h_{\lambda }$, add ${\mathbb{E}}_{q_{\lambda }}[{\nabla }_{\lambda }{h}_{\lambda }]$.

For a normalized differentiable sampling law on fixed support, ${\mathbb{E}}_q[\nabla \log q]=\nabla \int q=0$. Consequently any deterministic scalar $b$, independent of the sampled value, can be subtracted from the reward in a score-function gradient estimator without changing its expectation. This is an identity for the estimator, not a licence to differentiate an added parameter-dependent objective without its other terms. For a scalar parameter and score $S$ with $\mathbb{E}[S^2]>0$, minimizing the estimator's variance gives $b^{\ast }=\mathbb{E}[h{S}^2]/\mathbb{E}[S^2]$, provided these moments exist: expand $\mathbb{E}[(h-b{)}^2{S}^2]$ and differentiate the quadratic in $b$. An arbitrary baseline need not reduce variance.^[ref]