Scalloping loss

Scalloping loss is the underestimate of an isolated sinusoidal amplitude caused by measuring it only at the nearest sampled discrete Fourier transform frequency, after correcting the window’s coherent amplitude scale. An off-grid tone is evaluated away from the maximum of the window response. This differs from the coherent gain of a window, which describes the response at the correctly centered frequency. The instrument manual treats amplitude scaling and off-bin errors separately.^[ref]

Let $w_n$ be a finite real window with positive sum, and let $x_n=A{e}^{2\pi i(k+\delta )n/N}$, where $k$ is the nearest unpadded DFT bin and $|\delta |\le 1/2$. Write $W(\nu )={\sum }_{n=0}^{N-1}{w}_n{e}^{-2\pi in\nu }$. The coherent-gain-corrected bin magnitude relative to $|A|$ isHere $L$ is a positive loss when $r<1$. The half-bin amplitude response of a window evaluates this ratio at the largest nearest-bin offset. Other tones, noise and the negative-frequency image of a real sinusoid can also affect a measured peak; the derivation isolates one complex tone.

For a rectangular window, its geometric-sum transform givesFor $N=64$, the half-bin ratio is about $0.63668$ and the loss about $3.92\mathrm{dB}$. An amplitude-one tone exactly centered on a bin is recovered as one, while the same tone shifted by half a bin gives only about $0.637$ at either nearest bin. Dividing by coherent gain cannot remove this offset dependence.

For the periodic Hann window, usewhere $W_N$ is the rectangular response. At $N=64$, this gives a half-bin ratio about $0.84883$, or $1.42\mathrm{dB}$ loss. The smaller nearest-bin error comes with a wider main lobe and a larger equivalent noise bandwidth; window choice affects several measurement properties at once.

Zero padding reduces the nearest-grid offset without changing the measured record. Padding to $M=pN$ points makes the nearest offset at most $1/(2p)$ in original-bin units. With $p=4$ and $N=64$, the rectangular response at that worst offset is about $0.97450$, a loss of only $0.224\mathrm{dB}$. This is denser sampling of the same discrete-time Fourier transform, not extra physical frequency resolution. A fitted peak can reduce a grid error too, but its signal model and noise sensitivity must be stated.

A proposed analyzer test holds tone amplitude fixed while sweeping frequency through one bin interval, retaining record length, sampling rate and window. Divide out coherent gain, plot the measured peak versus fractional-bin offset, and compare with the exact response above. Include phase and nearby-tone checks for a real signal. Summing spectral powers instead answers an energy question and requires its own bandwidth normalization; it should not silently replace this single-bin amplitude definition.

The half-bin amplitude response of a window is the magnitude response to an isolated tone halfway between neighboring discrete Fourier transform frequencies, divided by the response to a bin-centered tone of the same amplitude. It quantifies a particular off-grid amplitude error after correcting the coherent gain of a window. For coefficients $w_0,\dots ,w_{N-1}$ with nonzero sum, defineA loss convention reports $L_{1/2}=-20{\log }_{10}{r}_{1/2}$ in decibels. The primary instrument manual tabulates approximately $3.92\mathrm{dB}$ for rectangular weighting and $1.42\mathrm{dB}$ for Hanning weighting.^[ref]

To derive the ratio, take a complex tone $x_n=A{e}^{2\pi i(k+1/2)n/N}$. Evaluating the windowed transform at bin $k$ gives $AW(-1/(2N))$. For real window coefficients, the responses at opposite offsets have equal magnitude. Dividing by $|W(0)|$ removes the on-bin scaling and leaves $|A|r_{1/2}$. This calculation isolates the frequency-grid effect from noise and other spectral components.

For a rectangular window, summing the geometric series yieldsFor $N=64$, the ratio is about $0.63668$. The large-$N$ value is about $0.63662$, explaining the familiar $3.92\mathrm{dB}$ scalloping loss rather than an exact finite-record constant.

For the periodic Hann window, $w_n=(1-\cos (2\pi n/N))/2$. Its finite response iswhere $W_N$ is the rectangular response and $W_H(0)=N/2$ for $N>1$. Substitution gives about $0.84883$ at $N=64$. A useful limiting derivation replaces the normalized sum by its integral:with continuous limiting values at $x=\pm 1$. At $x=1/2$ its magnitude is $8/(3\pi )\approx 0.84883$. For a short $N=4$ periodic window, the coefficients $(0,1/2,1,1/2)$ instead give $r_{1/2}=(1+\cos (\pi /4))/2\approx 0.85355$. The finite definition avoids treating the asymptotic value as exact.

The symmetric Hann window uses denominator $N-1$, not $N$; it has a different finite response and coefficient sum. SciPy explicitly distinguishes the symmetric filter-design convention from the periodic spectral convention.^[ref] The exact half-bin ratio should therefore use the coefficients actually applied.

For an original acquisition example, a $64$-sample record at $6400\mathrm{Hz}$ has $100\mathrm{Hz}$ bin spacing. A complex tone at $1050\mathrm{Hz}$ lies halfway between bins 10 and 11. An amplitude-one tone gives a corrected nearest-bin magnitude about $0.637$ with rectangular weighting or $0.849$ with periodic Hann weighting. Zero padding changes the sampled grid of the same discrete-time Fourier transform; if the padded grid contains $1050\mathrm{Hz}$, its value there has no half-bin error relative to that tone.

The half-bin response is not automatically a universal worst-case bound for every conceivable window. Nor does the isolated-complex-tone calculation cover every real-sinusoid measurement: near zero frequency or the sampling limit, the mirrored spectral component can overlap and make the observed magnitude phase-dependent. A proposed test sweeps a constant-amplitude tone through a bin interval, varies phase for real signals, and compares the complete finite-sum response before quoting a single off-bin accuracy number.