Solution (source code)

= Solution

Use $\widehat\psi_{\rm sc}(\nu)=\int\psi_{\rm sc}(x,0)e^{-i\nu x}dx$, inverse factor $1/(2\pi)$, and $k=\omega/c$. <Fourier transform> the homogeneous <Helmholtz equation> gives
$$
\partial_z^2\widehat\psi_{\rm sc}+(k^2-\nu^2)\widehat\psi_{\rm sc}=0.
$$
An upward radiating field has no incoming component from $z=+\infty$. For $q(\nu)=\sqrt{k^2-\nu^2}$, take the nonnegative real root when $|\nu|<k$ and the positive imaginary root when $|\nu|>k$. Consequently the <outgoing angular spectrum> is
$$
\boxed{\psi_{\rm sc}(x,z)=\frac1{2\pi}\int_{\mathbb R}
\widehat\psi_{\rm sc}(\nu)e^{i\nu x+iq(\nu)z}\,d\nu}.
$$
The propagating terms go upward; the <evanescent> terms decay with increasing $z$. Grazing values are understood by a radiation limit.

There is a reference-plane qualification in the question. For an arbitrary rough surface the full line $z=0$ need not lie in the medium: already $f(x)=h>0$ puts it in vacuum. In that case the stated physical trace is undefined without continuation. The exact representation is obtained instead on a plane $z=z_*>\sup f$,
$$
\psi_{\rm sc}(x,z)=\frac1{2\pi}\int
\widehat\psi_*(\nu)e^{i\nu x+iq(\nu)(z-z_*)}\,d\nu,\qquad z\ge z_*.
$$
One can formally set $\widehat\psi_{\rm sc}=\widehat\psi_*e^{-iqz_*}$, but convergence of that continued spectrum down to the rough boundary is an additional assumption. This is the <reference-plane validity of a rough-surface angular spectrum>. The first boxed formula is exact in an appropriate homogeneous upper half-space; it is not automatically an exact representation at every point of a completely arbitrary rough domain. Small-height <perturbation theory> in part (c) uses the mean plane order by order.