Gradient Estimation Using Stochastic Computation Graphs by John Schulman and others (2015, arXiv:1506.05254)

Primary conference paper. Section 3.1 and Appendix A derive the general gradient identity; Section 4 treats baselines. The construction combines random-choice probabilities and differentiable deterministic paths in a stochastic computation graph.