Autoregressive model ($\mathrm{AR}(p)$)

An autoregressive model expresses the current value as a linear combination of finitely many past values plus white noise.

For $X_t=\alpha X_{t-1}+{\alpha }^2{X}_{t-2}+{\epsilon }_t$, let $\phi =(1+\sqrt{5})/2$. The autoregressive polynomial factors as $(1-\alpha \phi z)(1+\alpha z/\phi )$. Its roots lie outside the closed unit disk exactly when $|\alpha |<1/\phi$. In that causal region,The coefficients follow from ${\psi }_0=1,{\psi }_1=\alpha$ and ${\psi }_j=\alpha {\psi }_{j-1}+{\alpha }^2{\psi }_{j-2}$. They are absolutely summable in the stated region. Causality makes future white noise orthogonal to the observation past, giving forecasts $\alpha X_T+{\alpha }^2{X}_{T-1}$ and $2{\alpha }^2{X}_T+{\alpha }^3{X}_{T-1}$ at horizons one and two.

For a causal autoregressive model $X_t={\sum }_{j=1}^p{\varphi }_j{X}_{t-j}+{ϵ}_t$ with centered white noise of variance ${\sigma }^2$, multiplying by past values and taking expectations yields $\gamma (h)={\sum }_j{\varphi }_j\gamma (h-j)$ for $h\ge 1$, and $\gamma (0)={\sum }_j{\varphi }_j\gamma (j)+{\sigma }^2$. These Yule-Walker equations determine the autocovariance from the coefficients and noise variance, or estimate coefficients from observed autocovariances. Causality supplies the needed orthogonality of the current noise to past observations.

For independent fair signs, $|X_n|\to \infty$ almost surely from every deterministic initial value. Outside $[-L,L]$ with $L>1/(a-1)$, the sign is preserved and $|X_{n+k}|\ge a^k(|X_n|-1/(a-1))+1/(a-1)$. To reach that region, choose $m$ so $(a^m-1)/(a-1)>L$. At the beginning of each $m$-step block, choose the sign of the current value; a block of that sign pushes the absolute value past $L$ and has conditional probability $2^{-m}$. The chance of remaining inside after $k$ blocks is at most $(1-2^{-m}{)}^k$. This proof avoids any assumption about a density of the limiting random series.

With two fixed initial states and iid N(0,1) errors, the parameter likelihood is the product of conditional transition densities. For $X_0=X_1=0$, it is proportional to $\exp [-\frac{1}{2}{\sum }_{t=0}^{n-2}(x_{t+2}-a{x}_{t+1}-b{x}_t{)}^2]$. The initial observation is a point mass and the first nonzero observation has a parameter-independent likelihood term.

The autoregressive polynomial is $\Phi (z)=1-{\sum }_{j=1}^p{\varphi }_j{z}^j$. Substituting the backshift operator gives the autoregressive filter. Zeros outside the unit disk give the causality root criterion for an autoregressive model; zeros on the unit circle obstruct a nondegenerate stationary innovation-driven solution.

The autoregressive operator acts as $\Phi (B)X_t=X_t-{\sum }_{j=1}^p{a}_j{X}_{t-j}$. Its coefficients are encoded by the autoregressive polynomial. An ARMA equation equates this filtered observation to the noise filtered by the moving-average operator.

A unit root is a zero of an autoregressive polynomial on the unit circle. A zero at $1$ is removed by first differencing. A conjugate pair $e^{\pm i\omega }$ corresponds to the real factor $1-2\cos \omega B+B^2$. Nonzero white-noise excitation at an uncancelled unit root prevents a finite-variance stationary solution.

A large, slowly damping oscillatory sample autocorrelation function suggests autoregressive zeros near a conjugate pair on the unit circle. For period $s$, the pair is near $e^{\pm 2\pi i/s}$, with minimal real filter $1-2\cos (2\pi /s)B+B^2$. Such a plot is evidence, not proof: a finite sample cannot distinguish a near-unit stationary model or a random sinusoid solely from the shape.

A causal time series satisfying $\Phi (B)X_t={\epsilon }_t$ with nondegenerate white noise has a square-summable present-and-past filter. The identity $\Phi (z)\Psi (z)=1$ excludes roots inside the unit disk; a pole of $1/\Phi$ on its boundary also prevents square summability. Thus all roots of $\Phi$ lie strictly outside the unit circle.

The inverse autoregressive polynomial is an analytic function on a disk larger than the unit disk. The Cauchy estimate gives exponentially decaying coefficients of its infinite moving-average representation. Summing their products gives $|\gamma (h)|\le C{s}^{|h|}$ for some $s\in (0,1)$.

A weakly stationary process satisfying an autoregressive model can depend on future driving white noise. For $X_t=\varphi X_{t-1}+{\epsilon }_t$ with $|\varphi |>1$, the convergent in the sense of mean-square convergence solution is $X_t=-{\sum }_{j\ge 1}{\varphi }^{-j}{\epsilon }_{t+j}$. Its autocovariance is ${\sigma }^2{\varphi }^{-|h|}/({\varphi }^2-1)$. This is different from a causal time series.

If an autoregressive model has white noise of positive variance, a unique weakly stationary process solving its equation on all integer times exists precisely when its autoregressive polynomial has no root on the unit circle. In that case $1/A(z)$ has an absolutely summable Laurent series on an annulus containing the unit circle; its coefficients define a bilateral linear filter of a stationary time series. Applying that inverse also proves uniqueness among bounded second-moment solutions. A root on the unit circle forces the candidate spectral density of a stationary process to have a nonintegrable singularity. The stronger causality root criterion for an autoregressive model requires every root outside the closed unit disk.

An autoregressive polynomial with a root at one gives a unit-root model. The basic example $X_t=X_{t-1}+{\epsilon }_t$ is a random walk, whose variance grows with time when the increments have positive variance. Differencing removes this root and recovers its white noise increments.

The Dickey–Fuller test detects a unit-root autoregressive process against a stationary causal alternative. Regress $\Delta X_t$ on $X_{t-1}$, with deterministic terms appropriate to the model. Under the null, the usual regression statistic has a nonnormal Brownian motion functional limit, so ordinary normal critical values are inappropriate.

A centered model $X_t={\varphi }_{\nu }{X}_{t-1}+{\sigma }_{\nu }{\epsilon }_t$, with $\nu =t\mathrm{mod}S$, has season-dependent coefficients and white noise innovations. In the nondegenerate case, the causal stability condition is $|{\prod }_{\nu =1}^S{\varphi }_{\nu }|<1$. Individual coefficients can exceed one in modulus while the product remains stable. Its second-order law is a periodically correlated process.

For unit-variance innovations, orthogonality to the past givesConsequently ${\varphi }_{\nu }={\gamma }_{\nu }(1)/V_{\nu -1}$ and ${\sigma }_{\nu }^2=V_{\nu }-{\gamma }_{\nu }(1{)}^2/V_{\nu -1}$. Replacing autocovariance by matched sample estimates yields seasonal regression estimators. The unit innovation variance is necessary to identify the noise scale separately.

An autoregressive process of order one obeyswhere $\varphi$ is real and ${\epsilon }_t$ is centered white noise of variance ${\sigma }^2>0$. The zero-mean version is $X_t=\varphi X_{t-1}+{\epsilon }_t$; in the affine form $X_t=c+\varphi X_{t-1}+{\epsilon }_t$, the stationary mean satisfies $c=(1-\varphi )\mu$.

For $|\varphi |<1$, its unique weakly stationary process solution on all integer times is the causal time-series representationThe sum converges in the sense of mean-square convergence because ${\sum }_{j\ge 0}{\varphi }^{2j}=1/(1-{\varphi }^2)<\infty$. It is an infinite moving-average representation using only present and past driving noise. Weak white noise suffices for this second-order conclusion; strong white noise also gives strict stationarity. The autocovariance of an AR(1) process is ${\sigma }^2{\varphi }^{|h|}/(1-{\varphi }^2)$, so its variance is ${\sigma }^2/(1-{\varphi }^2)$ and its ACF is ${\varphi }^{|h|}$. See Bartlett, lecture 5: AR(1) representation and causality.

The appeal of an AR(1) is that one coefficient expresses how a deviation persists. For $0<\varphi <1$, disturbances propagate with weights $1,\varphi ,{\varphi }^2,\dots$. For $-1<\varphi <0$, these weights alternate sign. With $\mu =10$, $\varphi =0.7$, and $X_t=12$, the recursive forecasts of an autoregressive process give $11.4$ and $10.98$ one and two steps ahead. If ${\sigma }^2=1$, the stationary variance is $100/51\simeq 1.9608$, while the one-step forecast error variance is only $1$. Under weak white noise these are best linear forecasts; independent driving noise also makes them conditional expectations given the observed past.

The condition $|\varphi |<1$ describes a stationary causal solution, not every finite recursion with those coefficients. Starting from a prescribed value gives the transient in finite initialization of an AR(1) process. Nor is it a condition for all possible two-sided stationary solutions: when $|\varphi |>1$, noncausal stationary autoregression uses future driving noise. See stationary versus causal solution of a two-sided AR(1) equation for the boundary cases.

For the stationary causal AR(1) with $|\varphi |<1$, matching terms in its infinite moving-average representation gives, for $h\ge 0$,Symmetry of the autocovariance extends this to negative lags. Dividing by $\gamma (0)$ gives the autocorrelation $\rho (h)={\varphi }^{|h|}$. The geometric series explains both the enlarged stationary variance and the diminishing memory: all past disturbances contribute to the present, but their squared weights have a finite sum. At $\varphi =0$, the process reduces to shifted white noise and all nonzero-lag autocorrelations vanish. See Bartlett, lecture 4: AR(1) representation and mean-square convergence.

For $t\ge 1$, iterating an AR(1) from a fixed value $X_0=x_0$ gives the displayed finite sum. With centered white noise of variance ${\sigma }^2$ and $|\varphi |<1$,For $h\ge 0$, its covariance is $\mathrm{Cov}(X_{t+h},X_t)={\varphi }^h\mathrm{Var}(X_t)$, which depends on $t$. Even setting $x_0=\mu$ leaves a changing variance. At $\varphi =0.7$, ${\sigma }^2=1$, and $x_0=\mu$, the variances at times zero through three are $0,1,1.49,1.7301$, approaching $100/51$; this finite initialized record is not weakly stationary.

To initialize in weak stationarity, give $X_0$ mean $\mu$, variance ${\sigma }^2/(1-{\varphi }^2)$, and zero covariance with every future driving noise value. The full stationary construction sets $X_0-\mu ={\sum }_{j\ge 0}{\varphi }^j{\epsilon }_{-j}$. For discrete Gaussian white noise, equivalently draw $X_0$ from $N(\mu ,{\sigma }^2/(1-{\varphi }^2))$ independently of future noise. This gives exact strict stationarity from the first plotted point; it is the initialization used in the figure in time series. Matching these first two moments alone does not guarantee strict stationarity for non-Gaussian inputs. See also stationary initialization of a scalar linear state-space model.

For $|\varphi |<1$, the bilateral sum ${\sum }_{k\in \mathbb{Z}}{\varphi }^{|k|}{z}^k$ equals $(1-{\varphi }^2)/((1-\varphi z)(1-\varphi z^{-1}))$ on the unit circle. Multiplying by ${\sigma }^2/(\pi (1-{\varphi }^2))$ gives the one-sided autoregressive spectrum and reads off its covariance coefficients.

For a causal AR(1) plus uncorrelated observation white noise, cross covariances vanish by L2 convergence of the autoregressive noise expansion. Its covariance is ${\sigma }_z^2{\varphi }^{|h|}/(1-{\varphi }^2)+{\sigma }_w^2{1}_{\{h=0\}}$. The additional noise changes only lag zero, attenuating the normalized positive-lag correlations.

Filtering the observations by $1-\varphi B$ gives covariance $A={\sigma }_z^2+(1+{\varphi }^2){\sigma }_w^2$ at zero, $C=-\varphi {\sigma }_w^2$ at lag one and zero elsewhere. Set $\nu =(A+\sqrt{A^2-4C^2})/2$ and $\vartheta =C/\nu$. Then $\nu (1+{\vartheta }^2)=A$, $\nu \vartheta =C$ and $|\vartheta |<1$. Applying $(1+\vartheta B{)}^{-1}$ defines actual white-noise innovations with variance $\nu$, proving an at-most-(1,1) ARMA representation without assuming Gaussianity.

With nondegenerate white noise, a two-sided AR(1) equation has a unique weakly stationary solution exactly when $|\varphi |\ne 1$. The solution is causal for $|\varphi |<1$ and anticausal for $|\varphi |>1$. At $\varphi =\pm 1$, an n-term noise sum has variance $n{\sigma }^2$, while its difference-of-stationary-values representation has variance at most four times the stationary variance, a contradiction.

For a stationary autoregressive process of order one with mean $\mu$, autoregressive coefficient $\varphi$ and innovation variance ${\sigma }^2$, an interior observation conditional on its neighbors hasThe Markov property makes the same law valid when all other observations are also conditioned on.

For $|\varphi |<1$, independent $N(0,{\sigma }^2)$ innovations and stationary initial variance ${\sigma }^2/(1-{\varphi }^2)$, the full likelihood function isConditioning on $X_1$ gives a different likelihood.

Conditioning on the first observation of a centered autoregressive process of order one makes the maximum-likelihood estimator of its coefficient the least-squares ratio above, when the optimum is interior. The stationary initial normal distribution contributes an additional factor $\sqrt{1-{\varphi }^2}$ and quadratic term $(1-{\varphi }^2)X_1^2$. These change the exact maximizer, but for a fixed interior stationary parameter their score is order one while the conditional score has order $T$, giving the same leading estimator.

For $X_t-\mu =\varphi (X_{t-1}-\mu )+{\epsilon }_t$, the autocovariance equation is $\gamma (1)=\varphi \gamma (0)$. Replacing both autocovariances by sample sums divided by the same $n$ yields ${\hat{\varphi }}_{YW}=\hat{\gamma }(1)/\hat{\gamma }(0)$, the lag-one sample autocorrelation function.

For a stationary unit-variance normal distribution autoregression with coefficient $a\in (-1,1)$, its vector of length $T\ge 2$ has precision matrix $J$ that is tridiagonal: the two endpoint diagonal entries are $1/(1-a^2)$, interior entries are $(1+a^2)/(1-a^2)$, and adjacent off-diagonal entries are $-a/(1-a^2)$. An independent observation $r=\lambda Z+\eta$, with covariance $vI$, updates precision and information vector toThe conditional law is multivariate normal distribution $N(Q^{-1}h,Q^{-1})$.