 Scalloping loss

Scalloping loss is the underestimate of an isolated sinusoidal amplitude caused by measuring it only at the nearest sampled discrete Fourier transform frequency, after correcting the window’s coherent amplitude scale. An off-grid tone is evaluated away from the maximum of the window response. This differs from the coherent gain of a window, which describes the response at the correctly centered frequency. The instrument manual treats amplitude scaling and off-bin errors separately.^[ref]

Let $w_n$ be a finite real window with positive sum, and let $x_n=A{e}^{2\pi i(k+\delta )n/N}$, where $k$ is the nearest unpadded DFT bin and $|\delta |\le 1/2$. Write $W(\nu )={\sum }_{n=0}^{N-1}{w}_n{e}^{-2\pi in\nu }$. The coherent-gain-corrected bin magnitude relative to $|A|$ isHere $L$ is a positive loss when $r<1$. The half-bin amplitude response of a window evaluates this ratio at the largest nearest-bin offset. Other tones, noise and the negative-frequency image of a real sinusoid can also affect a measured peak; the derivation isolates one complex tone.

For a rectangular window, its geometric-sum transform givesFor $N=64$, the half-bin ratio is about $0.63668$ and the loss about $3.92\mathrm{dB}$. An amplitude-one tone exactly centered on a bin is recovered as one, while the same tone shifted by half a bin gives only about $0.637$ at either nearest bin. Dividing by coherent gain cannot remove this offset dependence.

For the periodic Hann window, usewhere $W_N$ is the rectangular response. At $N=64$, this gives a half-bin ratio about $0.84883$, or $1.42\mathrm{dB}$ loss. The smaller nearest-bin error comes with a wider main lobe and a larger equivalent noise bandwidth; window choice affects several measurement properties at once.

Zero padding reduces the nearest-grid offset without changing the measured record. Padding to $M=pN$ points makes the nearest offset at most $1/(2p)$ in original-bin units. With $p=4$ and $N=64$, the rectangular response at that worst offset is about $0.97450$, a loss of only $0.224\mathrm{dB}$. This is denser sampling of the same discrete-time Fourier transform, not extra physical frequency resolution. A fitted peak can reduce a grid error too, but its signal model and noise sensitivity must be stated.

A proposed analyzer test holds tone amplitude fixed while sweeping frequency through one bin interval, retaining record length, sampling rate and window. Divide out coherent gain, plot the measured peak versus fractional-bin offset, and compare with the exact response above. Include phase and nearby-tone checks for a real signal. Summing spectral powers instead answers an energy question and requires its own bandwidth normalization; it should not silently replace this single-bin amplitude definition.