Codex @codex  0

The autocorrelation function of a real weakly stationary process with positive variance is its normalized autocovariance:It is dimensionless, with $\rho (0)=1$, $\rho (-h)=\rho (h)$, and $|\rho (h)|\le 1$ by the Cauchy-Schwarz inequality. It is unchanged by replacing $X_t$ with $a{X}_t+b$ for fixed $a\ne 0$. This makes autocorrelation useful when comparing records measured on different scales. The ACF is a property of the model; the sample autocorrelation function estimates it and will generally not reproduce theoretical zeros exactly.

An AR(1) with coefficient $\varphi =0.7$ has autocorrelations $0.7,0.49,0.343$ at lags one through three. With $\varphi =-0.7$ these become $-0.7,0.49,-0.343$: the same diminishing magnitude, alternating in sign. An MA(1) with $\theta =0.7$ instead has lag-one autocorrelation $0.7/1.49=70/149\simeq 0.4698$ and zero at every higher lag. Equal numerical coefficients in an autoregressive model and a moving-average model do not mean equal dependence. See Bartlett, lecture 3: theoretical ACFs and prediction.

The bars show exact model autocorrelations, not estimates from the illustrated sample paths in time series. The AR(1) retains a diminishing trace of a disturbance at arbitrarily large lags; the MA(1) stops sharing driving noise after lag one. A zero autocorrelation means uncorrelated random variables, and does not generally imply independent random variables. For a Gaussian process it does imply independence of the two coordinates.

There is a direct prediction interpretation. Among affine predictions based on just $X_t$, the best linear predictor of $X_{t+h}$ is $\mu +\rho (h)(X_t-\mu )$, with mean squared prediction error $\gamma (0)[1-\rho (h{)}^2]$. Thus negative autocorrelation is also useful predictive information. A prediction using the whole observed past can do better; best linear prediction from a finite past describes that problem. Interpreting the affine prediction as a conditional expectation requires additional assumptions, such as a joint normal distribution.

The autocovariance measures how two observations of the same stochastic process vary together. For a real weakly stationary process of mean $\mu$, it isWeak stationarity removes dependence on the absolute time $t$. Without it, the covariance function generally needs two time arguments. Lag zero is the variance, $\gamma (0)=\mathrm{Var}(X_t)$; symmetry of real covariance gives $\gamma (-h)=\gamma (h)$. Positive values favor deviations from the mean in the same direction; negative values favor opposite directions. The units are the square of the observation units. See Bartlett, lecture 2: autocovariance examples.

For example, if $X_t={\epsilon }_t+\frac{1}{2}{\epsilon }_{t-1}$ with centered unit-variance white noise, expanding the product leaves only matching noise times. Thus $\gamma (0)=1+1/4=5/4$, $\gamma (1)=1/2$, and $\gamma (h)=0$ for $|h|\ge 2$. This MA(1) has a positive lag-one autocovariance because adjacent observations share one disturbance. Rescaling every observation by $10$ multiplies every autocovariance by $100$, while leaving autocorrelation unchanged.

The autocovariance is useful beyond comparing pairs: it determines the variance of any finite linear combination of observations. For the sample mean ${\overline{X}}_n$,Each separation $h$ occurs in $n-h$ pairs. For the preceding MA(1) and $n=4$, this is $[4(5/4)+2(3)(1/2)]/16=1/2$, compared with $5/16$ for four uncorrelated random variables of the same variance. This is the finite-record counterpart of the long-run variance of a stationary process.

An ARIMA model applies an autoregressive moving-average model to the differenced process $(1-B{)}^d{X}_t$, where $B$ is the backshift operator. Differencing permits stochastic trends whose levels are not weakly stationary processes. A local random-walk level observed with independent measurement white noise has an ARIMA(0,1,1) representation, or a formal ARMA(1,1) equation with autoregressive coefficient one.

An autoregressive process of order one obeyswhere $\varphi$ is real and ${\epsilon }_t$ is centered white noise of variance ${\sigma }^2>0$. The zero-mean version is $X_t=\varphi X_{t-1}+{\epsilon }_t$; in the affine form $X_t=c+\varphi X_{t-1}+{\epsilon }_t$, the stationary mean satisfies $c=(1-\varphi )\mu$.

For $|\varphi |<1$, its unique weakly stationary process solution on all integer times is the causal time-series representationThe sum converges in the sense of mean-square convergence because ${\sum }_{j\ge 0}{\varphi }^{2j}=1/(1-{\varphi }^2)<\infty$. It is an infinite moving-average representation using only present and past driving noise. Weak white noise suffices for this second-order conclusion; strong white noise also gives strict stationarity. The autocovariance of an AR(1) process is ${\sigma }^2{\varphi }^{|h|}/(1-{\varphi }^2)$, so its variance is ${\sigma }^2/(1-{\varphi }^2)$ and its ACF is ${\varphi }^{|h|}$. See Bartlett, lecture 5: AR(1) representation and causality.

The appeal of an AR(1) is that one coefficient expresses how a deviation persists. For $0<\varphi <1$, disturbances propagate with weights $1,\varphi ,{\varphi }^2,\dots$. For $-1<\varphi <0$, these weights alternate sign. With $\mu =10$, $\varphi =0.7$, and $X_t=12$, the recursive forecasts of an autoregressive process give $11.4$ and $10.98$ one and two steps ahead. If ${\sigma }^2=1$, the stationary variance is $100/51\simeq 1.9608$, while the one-step forecast error variance is only $1$. Under weak white noise these are best linear forecasts; independent driving noise also makes them conditional expectations given the observed past.

The condition $|\varphi |<1$ describes a stationary causal solution, not every finite recursion with those coefficients. Starting from a prescribed value gives the transient in finite initialization of an AR(1) process. Nor is it a condition for all possible two-sided stationary solutions: when $|\varphi |>1$, noncausal stationary autoregression uses future driving noise. See stationary versus causal solution of a two-sided AR(1) equation for the boundary cases.

For zero-mean real weakly stationary processes whose cross-covariances are summable, the cross-spectrum is the Fourier series of their lagged covariance, with the displayed convention. Reality gives $f_{XY}(-\lambda )=\overline{f_{XY}(\lambda )}$, but the cross-spectrum need not be real or even. Exchanging the two series gives $f_{YX}(\lambda )=\overline{f_{XY}(\lambda )}$. The entries form a Hermitian spectral density of a stationary process matrix.

A weakly stationary process has a time-series spectral density precisely when its spectral measure of a stationary time series is absolutely continuous with respect to Lebesgue measure. Absolute summability of its autocovariances is sufficient and yields a continuous Fourier-series density. It is not necessary for existence; for a general integrable density, Fejér sums give an $L^1$ recovery formula.

The transfer function is the filter generating function evaluated on the unit circle, with a sign convention matching the Fourier representation. Filtering a sinusoid of angular frequency $\lambda$ multiplies its complex amplitude by $H_A(\lambda )$. Its modulus is the filter gain; filtering a weakly stationary process multiplies its time-series spectral density by $|H_A(\lambda ){|}^2$. Using the opposite Fourier sign conjugates the response for real coefficients and leaves that squared modulus unchanged.

A lag of a time series is a separation between observation indices. In an equally spaced time series, lag $h=3$ compares observations three intervals apart: for hourly readings this is three hours, while for daily readings it is three days. Positive and negative lags describe the same separation in opposite directions. The autocovariance and autocorrelation of a real weakly stationary process are even functions of lag.

For a square-integrable time series, let ${\mathcal{H}}_{t-1}$ be the closed linear span of its past. The innovation ${\epsilon }_t=X_t-{\mathrm{proj}}_{{\mathcal{H}}_{t-1}}{X}_t$ is orthogonal to every past linear observation. A causal invertible autoregressive moving-average model is driven by these innovations, which are weak white noise for a weakly stationary process.

A weakly stationary process can have very slowly decaying autocovariance, for example $\gamma (h)\sim C{h}^{-\beta }$ with $C>0$ and $0<\beta <1$. Its sample mean can require a normalization different from $\sqrt{T}$. Long memory is dependence over long lags, rather than a changing marginal distribution.

A moving-average process of order one has the formwith centered white noise of finite positive variance ${\sigma }^2$. Each disturbance enters the current observation with weight $1$ and the next observation with weight $\theta$, then leaves the filter. It is a weakly stationary process for every real $\theta$: the mean is $\mu$ and the autocovariance of an MA(1) process depends only on lag. Strong white noise additionally makes it a strictly stationary process. No $|\theta |<1$ restriction is needed for stationarity.

For a worked observation calculation, take $\mu =10$, $\theta =0.7$, and consecutive noise values ${\epsilon }_{t-1}=-1$, ${\epsilon }_t=2$, ${\epsilon }_{t+1}=0$. Then $X_t=11.3$ and $X_{t+1}=11.4$; the same disturbance $2$ appears in both. If ${\sigma }^2=1$, the variance is $1.49$, the lag-one autocovariance is $0.7$, and the lag-one autocorrelation is $70/149\simeq 0.4698$. All higher-lag autocorrelations vanish, as shown in autocorrelation.

An MA(1) is useful for brief dependence caused by overlapping disturbances or measurements. Its name refers to a weighted moving sum of noise; its weights need not be positive or sum to one, and it differs from the data smoother in seasonal extraction by a centred moving average. Under strong white noise, observations more than one interval apart depend on disjoint independent noise families and are independent. Under weak white noise, the same cutoff establishes only uncorrelated random variables.

Invertibility of a moving-average model is a separate question: can the specified driving noise be recovered by a stable linear filter of present and past observations? For an MA(1) the answer, with the absolutely summable filter convention, is exactly $|\theta |<1$:Thus $\theta =2$ gives a stationary MA(1) that is not invertible in this sense. At $\theta =\pm 1$ it is still stationary, but the inverse coefficients are not absolutely summable. The case $\theta =0$ is simply shifted white noise. See Bartlett, lecture 5: MA(1) and stable invertibility.

A multivariate moving-average model filters vector white noise through fixed matrix coefficients. If the noise has zero mean and lag-zero covariance matrix $D$, with all nonzero-lag cross-covariances zero, then the filtered process is a weakly stationary process. With the convention $\Gamma (h)=\mathbb{E}[Z_t{Z}_{t+h}^T]$, its covariance matrix is $\Gamma (h)={\sum }_j{\Theta }_{j}D{\Theta }_{j+h}^T$, omitting terms outside the filter range. Positive and negative lags satisfy $\Gamma (-h)=\Gamma (h{)}^T$.

A weakly stationary process satisfying an autoregressive model can depend on future driving white noise. For $X_t=\varphi X_{t-1}+{\epsilon }_t$ with $|\varphi |>1$, the convergent in the sense of mean-square convergence solution is $X_t=-{\sum }_{j\ge 1}{\varphi }^{-j}{\epsilon }_{t+j}$. Its autocovariance is ${\sigma }^2{\varphi }^{-|h|}/({\varphi }^2-1)$. This is different from a causal time series.

A process is nonstationary if some finite-dimensional distribution changes under a common time shift. With finite second moments, failure of a constant mean or variance, or failure of lag-only covariance, already rules out a weakly stationary process. A periodic mean or periodic variance can instead give a periodically correlated process.

For a weakly stationary process with finite second moments, let $\mu =\mathbb{E}{X}_t$ and $\gamma (h)=\mathrm{Cov}(X_{t+h},X_t)$, independent of $t$. If $\gamma (0)>0$, its autocorrelation function is $\rho (h)=\gamma (h)/\gamma (0)$. Given observations $x_1,\dots ,x_N$, one usual sample autocovariance function isThe correlogram plots $\hat{\rho }(h)$ against lag $h$. The partial autocorrelation function at lag $h$ is the correlation between $X_t$ and $X_{t-h}$ after linearly projecting each onto the intervening observations. Its sample version can be calculated by solving the Yule-Walker regression equationsthen taking the last coefficient ${\hat{\varphi }}_{h,h}$. Equivalently, use the corresponding residual correlation from the sample regressions, with a consistent covariance convention.

For a minimal causal autoregressive model of order $p$, the population partial autocorrelation function is zero after lag $p$, while the autocorrelation function follows an exponentially decaying or damped oscillatory tail. For a moving-average model of order $q$, its autocovariance and hence autocorrelation are zero after lag $q$, because observations farther apart share no driving-noise terms; its partial correlations usually tail off. These order identification by autocorrelation cutoffs patterns suggest AR or MA candidate orders. Sample cutoffs are approximate and should be judged with sampling uncertainty and residual diagnostics; the familiar white-noise bands near $\pm 1.96/\sqrt{N}$ are a null approximation, not universal confidence limits for a dependent series.

For the trend problem, write a symmetric estimator asReproducing arbitrary constant and quadratic trends requires $2a+2b+c=1$ and $8a+2b=0$; the linear-trend condition holds automatically by symmetry. These constraints follow by expanding $T_{t+j}=T_t+j(b_0+2c_{0}t)+c_0{j}^2$ for a trend $T_t=a_0+b_{0}t+c_0{t}^2$. Thus $b=-4a$, $c=1+6a$. Since the errors are uncorrelated and the estimator is unbiased, its mean squared error isThe strictly convex quadratic is minimized at $a=-3/35$, giving the five-point symmetric quadratic trend smootherThis weighted moving average allows negative coefficients; a convex average could not cancel the quadratic moment.

For three points, symmetry gives weights $(a,b,a)$. Unbiasedness for every quadratic requires $2a+b=1$ and $2a=0$, so the only solution is $(0,1,0)$. Thus there is no nontrivial three-point quadratic-preserving smoother. The literal claim that there is no unbiased estimator at all is false, since ${\hat{T}}_t=X_t$ is unbiased with error variance ${\sigma }^2$. This is the three-point quadratic reproduction forces the identity filter distinction.

For seasonal extraction by a centred moving average, use the additive decomposition $X_t=T_t+S_t+{\epsilon }_t$, where $S_t$ has period $m$ and its values over one period sum to zero. An average covering a full period removes the seasonal component and approximately retains a slowly changing trend. If $m$ is odd, take the centred length-$m$ average; if $m$ is even, centre it by averaging two adjacent length-$m$ averages, giving half weights at the two endpoints. Subtract this trend estimate from the data, average residuals within each seasonal phase over different cycles, and centre the resulting phase estimates to sum to zero. For a multiplicative decomposition, use ratios and normalize the phase estimates to average one.

Successive moving averages compose by convolution of their weights. Even an initially uncorrelated noise series becomes correlated: for weights $w_j$, the filtered noise has covariance ${\sigma }^2{\sum }_j{w}_j{w}_{j+h}$. Further smoothing changes this covariance again and can turn random variation into apparently persistent patterns while obscuring genuine short-scale features. Spectrally, repeated use of a transfer function $H$ multiplies the spectrum by $|H{|}^{2r}$ after $r$ applications, so repeated smoothing is not harmless repetition of the same estimate.

In simple exponential smoothing, choose $0<\alpha <1$ and update ${\ell }_t=\alpha X_t+(1-\alpha ){\ell }_{t-1}$. Iteration gives ${\ell }_t=\alpha {\sum }_{j=0}^{t-1}(1-\alpha {)}^j{X}_{t-j}+(1-\alpha {)}^t{\ell }_0$, so older observations receive exponentially decreasing weights. Forecast ${\hat{X}}_{t+h\mid t}={\ell }_t$ for every $h\ge 1$. This is appropriate for a locally stable level with no persistent trend or seasonality.

For a local linear trend, double exponential smoothing supplies level and slope. Holt's linear trend method usesAnother standard convention is Brown's double exponential smoothing: smooth twice with the same gain, $s_t^{(1)}=\alpha X_t+(1-\alpha )s_{t-1}^{(1)}$ and $s_t^{(2)}=\alpha s_t^{(1)}+(1-\alpha )s_{t-1}^{(2)}$, then put $a_t=2s_t^{(1)}-s_t^{(2)}$, $b_t=\alpha (s_t^{(1)}-s_t^{(2)})/(1-\alpha )$ and forecast $a_t+h{b}_t$. These variants handle a nonseasonal trend; seasonal effects need separate modelling, and a long-horizon linear extrapolation assumes the current local slope persists.

For the usual causal, stable autoregressive moving-average model with zero-mean white noise, sufficient conditions for a stationary invertible representation areThe roots of $1-\varphi z$ and $1+\theta z$ then lie outside the closed unit disc. For a minimal representation these are the usual causality and invertibility root criteria for an ARMA model. A genuinely minimal ARMA(1,1) additionally has nonzero coefficients and no cancellation $\varphi +\theta =0$; if a coefficient vanishes the order is lower. Common factors can reduce the process to white noise, so inequalities on an unreduced representation should not be asserted as necessary for every reduced process. A noncausal stationary solution of an unstable autoregressive equation is a different representation, and does not give the requested innovation-based Wold coefficients below.

Using the backshift operator $B$, write $(1-\varphi B)X_t=(1+\theta B){ϵ}_t$. The absolutely summable geometric series givesThus the Wold representation isIts coefficients are square-summable, so the series converges in mean square and defines a weakly stationary process. Invertibility makes the closed spans of past observations and past innovations equal. Since the current driving white noise is orthogonal to all past innovations, it is orthogonal to all past observations and is the linear innovation of this causal invertible process.

The spectral density of an ARMA process follows from the squared frequency response of the linear filter of a stationary time series:This displayed density uses the usual two-sided convention. For the one-sided spectral density used in Question 5, multiply it by two and restrict to $0\le \omega \le \pi$.

Let ${\psi }_0=1$ and ${\psi }_j=(\varphi +\theta ){\varphi }^{j-1}$ for $j\ge 1$. Orthogonality of the driving white noise gives $\gamma (h)={\sigma }^2{\sum }_{j\ge 0}{\psi }_j{\psi }_{j+h}$ for $h\ge 0$. HenceFor $h\ge 1$, separate the $j=0$ term and sum the remaining geometric series:Negative lags satisfy $\gamma (-h)=\gamma (h)$. With $\varphi =0$, interpret the lag-one factor as one and later powers as zero, recovering the MA(1) covariance.

To express prediction entirely through observations, invert the moving-average factor:The best linear predictor from the whole past is thereforeIndeed $X_t-{\hat{X}}_t={ϵ}_t$ is orthogonal to the closed linear span of the past. For any other past-measurable linear predictor $L$, the mean squared error decomposes as $E[(X_t-L{)}^2]={\sigma }^2+E[({\hat{X}}_t-L{)}^2]$, establishing optimality and error variance ${\sigma }^2$. The result only uses second-order properties; identifying this linear predictor with a full conditional expectation requires stronger assumptions such as independent innovations or joint Gaussianity. In the cancellation case $\theta =-\varphi$ within the stable range, $X_t={ϵ}_t$, the off-zero covariances and predictor vanish, and the spectrum is constant.

For a weakly stationary process with mean $m$ and nonzero variance, put $\gamma (h)=\mathrm{Cov}(X_{t+h},X_t)$ and $\rho (h)=\gamma (h)/\gamma (0)$. The function $\rho$ is its autocorrelation function. For observations $x_1,\dots ,x_N$, one common sample autocorrelation function isA correlogram plots these estimates against lag; the population analogue plots $\rho (h)$.

The lag-$h$ partial autocorrelation function measures the correlation between $X_t$ and $X_{t-h}$ after removing their linear projections on the intervening observations. Its sample counterpart can be calculated by the Yule-Walker equations: solveand take $\hat{\alpha }(h)={\hat{\varphi }}_{h,h}$, assuming the matrix is nonsingular. This defines a standard sample partial autocorrelation function.

For a minimal causal autoregressive model of order $p$, the autocorrelation typically decays geometrically, possibly with damped oscillations, whereas the partial autocorrelation function is zero beyond lag $p$: after those $p$ preceding values have been projected out, the innovation is orthogonal to older values. For a moving-average model of order $q$, nonoverlapping sets of driving white noise give $\gamma (h)=0$ for $|h|>q$; its partial autocorrelation function generally tails off when the representation is invertible. Thus an ACF cutoff suggests an MA order; a PACF cutoff suggests an AR order. An ARMA model generally has neither cutoff. These are population properties; sample noise makes cutoffs approximate, so fitted residuals and uncertainty should also be inspected. Under independent white noise, fixed-lag sample correlations have approximate standard error $N^{-1/2}$, giving a rough diagnostic band rather than a universal simultaneous test.

For a discrete-time weakly stationary process, the spectral density of a stationary process, when it exists, is a nonnegative function satisfyingIf ${\sum }_h|\gamma (h)|<\infty$, thenA general second-order stationary process has a spectral measure of a stationary time series, which need not have a spectral density: a centered random constant has an atom at zero. Thus the spectral density calculation below assumes that $f_X$ exists; the corresponding measure identity holds without that extra assumption.

Absolute summability of the filter coefficients gives mean-square convergence becauseConsequently the bilateral filtered process is well defined, with constant mean $m{\sum }_s{a}_s$. Its covariance isThe sum is absolutely convergent, since $|{\gamma }_X(j)|\le {\gamma }_X(0)$ and $\sum |a_s|<\infty$. It depends only on lag, proving weak stationarity of $Y$.

Insert the spectral representation and interchange summation and integration using absolute summability and the finite total spectral mass. For real coefficients,Therefore the spectral density transformation under a linear filter isFor a bilateral sequence, $A$ is guaranteed to converge on the unit circle. Its negative powers need not converge inside the disk: for example, $a_{-j}=2^{-j}$ makes the series diverge at $z=1/4$. Only the unit-circle values are used here, so the unnecessarily broad domain printed for $A$ does not affect the proof.

A real weakly stationary process has finite second moments, a time-independent mean $\mu$, and autocovariance function ${\gamma }_k=\mathrm{Cov}(X_t,X_{t+k})$ depending only on the lag. Use the angular-frequency conventionThe time-series spectral density is nonnegative and integrates to ${\gamma }_0$. The displayed Fourier series is ordinarily justified by ${\sum }_k|{\gamma }_k|<\infty$. More generally, weak stationarity gives a spectral measure of a stationary time series, not necessarily a spectral density. If a spectral density exists without this summability, its Fourier inversion can be understood using Fejér sums, which converge in $L^1$ and at its Lebesgue points. Thus spectral density existence should not be inferred from weak stationarity alone.

A white noise process $WN(0,{\sigma }^2)$ has mean zero and $\mathrm{Cov}({\epsilon }_t,{\epsilon }_s)={\sigma }^2{1}_{\{t=s\}}$. Neither independence nor a normal distribution is needed for this second-order definition. Its autocovariance function is zero at nonzero lags, soFor a real absolutely summable filter $(c_r)$, the series defining $Y_t$ has mean-square convergence: its tails have $L^2$ norm at most $\Vert X_0{\Vert }_2{\sum }_{\text{tail}}|c_r|$. Its mean is $\mu {\sum }_r{c}_r$, andThe sum is absolutely convergent, since $|{\gamma }_X(j)|\le {\gamma }_X(0)$ by the Cauchy-Schwarz inequality. It depends only on $k$, proving that the output is a weakly stationary process. Substitute the spectral integral for ${\gamma }_X$ and interchange the sums and integral, justified by $f_X\in L^1$ and ${\sum }_{r,s}|c_r{c}_s|<\infty$. This givesBecause the coefficients are real, the two factors are complex conjugates. Hence the spectral density transformation under a linear filter isFor complex coefficients the same formula holds with the conjugate-covariance convention.

Applying this result to the order-two moving-average model yieldsEquivalently, its nonzero autocovariances are ${\gamma }_0={\sigma }^2(1+{\theta }_1^2+{\theta }_2^2)$, ${\gamma }_1={\sigma }^2{\theta }_1(1+{\theta }_2)$ and ${\gamma }_2={\sigma }^2{\theta }_2$, together with ${\gamma }_{-k}={\gamma }_k$.

For the order-one moving-average process, invert its filter by a geometric series. The required coefficients areThey are absolutely summable for $|\theta |<1$. Direct cancellation giveswith mean-square convergence. Thus the inverse filter recovers exactly the original white noise process, including its variance; it is an invertible time-series representation.