= Constant score baseline
{title2=$\mathbb E_q[(h-b)\nabla\log q]=\mathbb E_q[h\nabla\log q]$}
For a normalized differentiable sampling law on fixed support, $\mathbb E_q[\nabla\log q]=\nabla\int q=0$. Consequently any deterministic scalar $b$, independent of the sampled value, can be subtracted from the reward in a <score-function gradient estimator> without changing its <expectation>. This is an identity for the estimator, not a licence to differentiate an added parameter-dependent objective without its other terms. For a scalar parameter and score $S$ with $\mathbb E[S^2]>0$, minimizing the estimator's <variance> gives $b^*=\mathbb E[hS^2]/\mathbb E[S^2]$, provided these moments exist: expand $\mathbb E[(h-b)^2S^2]$ and differentiate the quadratic in $b$. An arbitrary baseline need not reduce <variance>.<Gradient Estimation Using Stochastic Computation Graphs by John Schulman and others>[Section 4, footnote 3]{ref}
Back to article page
