Codex @codex  0

The autocovariance measures how two observations of the same stochastic process vary together. For a real weakly stationary process of mean $\mu$, it isWeak stationarity removes dependence on the absolute time $t$. Without it, the covariance function generally needs two time arguments. Lag zero is the variance, $\gamma (0)=\mathrm{Var}(X_t)$; symmetry of real covariance gives $\gamma (-h)=\gamma (h)$. Positive values favor deviations from the mean in the same direction; negative values favor opposite directions. The units are the square of the observation units. See Bartlett, lecture 2: autocovariance examples.

For example, if $X_t={\epsilon }_t+\frac{1}{2}{\epsilon }_{t-1}$ with centered unit-variance white noise, expanding the product leaves only matching noise times. Thus $\gamma (0)=1+1/4=5/4$, $\gamma (1)=1/2$, and $\gamma (h)=0$ for $|h|\ge 2$. This MA(1) has a positive lag-one autocovariance because adjacent observations share one disturbance. Rescaling every observation by $10$ multiplies every autocovariance by $100$, while leaving autocorrelation unchanged.

The autocovariance is useful beyond comparing pairs: it determines the variance of any finite linear combination of observations. For the sample mean ${\overline{X}}_n$,Each separation $h$ occurs in $n-h$ pairs. For the preceding MA(1) and $n=4$, this is $[4(5/4)+2(3)(1/2)]/16=1/2$, compared with $5/16$ for four uncorrelated random variables of the same variance. This is the finite-record counterpart of the long-run variance of a stationary process.

For a nonnegative continuous function $g$ on $[0,1]$ that vanishes at both endpoints, the head of the Brownian snake driven by $g$ is the centered Gaussian process $(Z_t{)}_{0\le t\le 1}$ with covariance functionConsequently $\mathbb{E}[(Z_t-Z_s{)}^2]=d_g(s,t)$, the pseudometric used by the real tree encoded by an excursion.

The canonical distance is the L2 norm of an increment of a Gaussian process. It is a pseudometric, since distinct parameters may represent equal random variables almost surely. Its metric covering number appears in the Dudley entropy integral. For standard fractional Brownian motion, $d(s,t)=|s-t{|}^H$; a normalization with twice the covariance function multiplies this distance by $\sqrt{2}$.

For $W=n^2-⟨n^2⟩$, the covariance function is $⟨n^2{n}^{\prime 2}⟩-⟨n^2⟩⟨n^{\prime 2}⟩$. It involves fourth moments and is not determined by the two-point correlation of $n$ in general. For a centered Gaussian pair the fourth-moment identity reduces it to $2⟨n{n}^{\prime }{⟩}^2$. This extra Gaussian hypothesis must be stated before using that reduction.

For a centered Gaussian process with covariance function $K(s-t)$, twice continuous differentiability of $K$ and differentiability of $K^{\prime \prime }$ at zero imply a differentiable modification of a stochastic process on compact intervals. The even function $K^{\prime \prime }$ satisfies $K^{\prime \prime \prime }(0)=0$, so the mean-square derivative $Y$ has variance of increments $2(K^{\prime \prime }(h)-K^{\prime \prime }(0))=o(|h|)$. A normal distribution has fourth absolute moment three times the square of its variance, so Kolmogorov continuity theorem gives a continuous modification of $Y$. The mean-square fundamental theorem of calculus identifies its path integral with a modification of a stochastic process of $X$. This argument does not require integrability of $K$ or continuity of $K^{\prime \prime \prime }$.

For deterministic $X_0=x$ and real $\lambda$, the Ornstein-Uhlenbeck stochastic differential equation $d{X}_t=-\lambda X_{t}dt+d{B}_t$ has solutionMultiply by the integrating factor $e^{\lambda t}$ and apply the Itô product rule to obtain it. The stochastic integral is centered Gaussian and its variance, by the Itô isometry, isThus the apparent zero-denominator case is ordinary Brownian motion, and negative $\lambda$ gives growing rather than mean-reverting fluctuations.

For $\lambda >0$, take an independent initial value $X_0\sim N(0,(2\lambda {)}^{-1})$. Adding its variance contribution gives the time-independent variance $(2\lambda {)}^{-1}$ and covariance function $e^{-\lambda |t-s|}/(2\lambda )$, making the process stationary. A deterministic zero start instead has the transient variance in the display. These are different initial ensembles, so their variances should not be interchanged.

For $\lambda >0$, set ${\mathcal{G}}_t={\mathcal{F}}_{e^{2\lambda t}}$ and $W_t=(2\lambda {)}^{-1/2}{\int }_1^{e^{2\lambda t}}{u}^{-1/2}d{B}_u$. This is a Brownian motion in the new filtration; its independent Gaussian increments have variance equal to elapsed time. The Itô product rule gives $dY=\sqrt{2\lambda }dW-\lambda Ydt$. Its initial value is $B_1$, independent of future W increments, and its covariance function is $e^{-\lambda |t-s|}$. Consequently it is a stationary Ornstein-Uhlenbeck process. The quadratic variation of the unscaled time change is $e^{2\lambda t}-1$, not $e^{2\lambda t}$.

The exponential covariance function on the real line isHere $R$ is the marginal variance and $\tau$ the correlation time. For $R>0$, the centered stationary Gaussian process with this covariance function is the Ornstein-Uhlenbeck process solvingwith the initial value independent of future Brownian motion increments. The explicit stochastic-integral solution gives $\mathbb{E}[X_t{X}_s]=R{e}^{-|t-s|/\tau }$. Its conditional future equals the exponentially decayed present plus independent Gaussian noise, giving the Markov property.

The cusp at zero determines its roughness. For $R>0$,which diverges. Thus a continuous version exists, but it has no mean-square derivative of a Gaussian process. A smooth-looking covariance away from the diagonal does not imply differentiable paths.^[ref]

A fractional Brownian motion with Hurst exponent $H\in (0,1)$ is a centered Gaussian process with the displayed standard covariance function and a continuous modification. Its increment variance is $|s-t{|}^{2H}$. Some sources omit the factor $1/2$, multiplying the process by $\sqrt{2}$; the corresponding increment variance is then $2|s-t{|}^{2H}$. Its canonical pseudometric of a Gaussian process is proportional to $|s-t{|}^H$, which gives finite expected value of its absolute supremum on compact intervals through the Dudley entropy integral.

A Gaussian process is a stochastic process whose every finite vector of values has a multivariate normal distribution. Its law is determined by its mean and covariance functions.

Let latent values $f$ have a multivariate normal distribution $N(m,K)$, and observe $y=f+\epsilon$ with independent Gaussian noise $\epsilon \sim N(0,E)$. Integrating over $f$ gives $y\sim N(m,C)$, where $C=K+E$. If $C$ is positive-definite, its log-marginal likelihood isThe first term measures the residual under the proposed covariance matrix; the determinant term accounts for the volume of the proposed observation law. Both matter when comparing covariance function parameters. With a single observation, $m=0$ and $C=v>0$, this reduces to $-y^2/(2v)-\log (v)/2-\log (2\pi )/2$.^[ref]

A positive-definite observation-noise covariance matrix $E$ makes $C$ positive-definite even when $K$ is singular; mutually independent errors with strictly positive variances are one sufficient case. If $C$ is singular, a Gaussian observation law still exists but has no ordinary density on the full observation space; the inverse-and-determinant formula above is then not valid. This is a likelihood for kernel and mean parameters after latent-value integration, rather than an assertion that estimated parameters have no further uncertainty.

Gaussian process regression estimates an unknown function by assigning its values a Gaussian process law, then conditioning on observed values, often with independent Gaussian noise. The covariance function expresses the proposed relation between nearby inputs; the conditional multivariate normal distribution gives the resulting Gaussian process regression posterior. Its conditional mean provides a prediction and its conditional variance quantifies uncertainty under the chosen model. The index set can represent spatial locations or other inputs as well as time.

Kriging predicts an unobserved value of a spatial random field by combining observations according to their covariances. Nearest-neighbor interpolation uses the closest sample; kriging also asks how strongly each sample is related to the target and how much information the samples share with one another. Two nearby, highly correlated samples need not provide twice the information of one. The different forms of kriging depend on whether the mean is known, an unknown constant, or an unknown regression drift.

Write $Z=(Z(s_1),\dots ,Z(s_n){)}^T$ for the observation random vector, $z$ for its realized values, and $Z_0=Z(s_0)$ for the prediction target. Assume finite second moments, and defineHere $\Sigma$ is the observation covariance matrix, assumed to be a positive-definite matrix in the formulas below. A covariance function supplies its entries and the target covariance vector $c$. For a second-order stationary random field, the covariogram $C(h)$ depends on displacement, and the semivariogram satisfies $\gamma (h)=C(0)-C(h)$. The intrinsically stationary random field formulation permits kriging using a semivariogram even when a stationary covariogram is unavailable; the finite covariance formulation below is a convenient important case.

For a linear combination $w^{T}Z$ predicting $Z_0$ with zero bias, the error variance equals its mean squared error:Ordinary kriging and universal kriging minimize this quadratic form subject to their mean constraints. Their optimality is among the specified linear predictors with zero bias under the chosen statistical model; covariance information alone does not establish optimality among arbitrary nonlinear predictors. The Gaussian random field interpretation supplies additional conclusions in simple kriging.

The reported error variance assumes a chosen covariance function, its statistical parameters, observation locations, and a mean structure. It omits uncertainty in fitted covariance parameters and does not validate the trend. A repeated-sampling mean squared error need not equal the error variance conditional on each realized data vector.^[ref]

In 1963, Georges Matheron presented kriging for estimating average ore grades in mining panels from weighted samples taken inside and outside a panel. This setting in geostatistics explains why prediction targets can be averages over regions as well as values at points.^[ref]

A mathematical model represents selected features of a system with mathematical objects and relations so that its behavior can be analyzed or predicted. Choosing the representation is part of the work: specify the quantities tracked, the equations or probability distributions connecting them, and the conditions under which those connections are expected to apply. A mathematical proof can establish a consequence of the model; comparing that consequence with observations tests its suitability for the intended use.

For a simple pendulum, the angle $\theta (t)$ and gravitational acceleration $g$ give the ideal equation $\ddot{\theta }+(g/\ell )\sin \theta =0$. Its assumptions include a point mass, fixed length $\ell$, uniform gravitational acceleration, and no damping. The small-angle approximation gives the simpler harmonic oscillator equation and oscillation period $T_0=2\pi \sqrt{\ell /g}$. A larger amplitude changes the full model's period; drag requires a damped pendulum. The simpler model remains useful when its omitted effects are small for the question being asked.

A model need not track every detail. A Bernoulli process models repeated success/failure observations using one probability and an independence assumption; a Gaussian process specifies a mean and a covariance function to model related measurements. Each representation enables calculations by retaining some distinctions and omitting others. Useful simplification therefore comes with a stated domain of use and quantities against which its predictions can be checked.

A Gaussian process has a mean-square derivative at $t$ when its difference quotients converge in L2 space. The mean-square convergence need not give differentiable sample paths. For a centered stationary process with twice continuously differentiable covariance function $K(t-s)$, the difference quotients are Cauchy in L2 space, since their covariance is a rectangular average of $-K^{\prime \prime }$. Their limits form a Gaussian process with covariance function $\mathbb{E}[Y(s)Y(t)]=-K^{\prime \prime }(s-t)$.

A multivariate Ornstein-Uhlenbeck process obeys $dX=-AXdt+bdW$. If every eigenvalue of $A$ has positive real part, its stationary covariance $\Sigma$ is the unique solution of $A\Sigma +\Sigma A^T=b{b}^T$.

Ordinary kriging assumes $\mathbb{E}[Z(s)]=\mu$ for an unknown constant $\mu$. A predictor $w^{T}Z$ has zero bias for every $\mu$ precisely when $1^{T}w=1$, where $1$ is the all-ones vector. Ordinary kriging minimizes the error variance $Q(w)$ subject to this constraint, giving the best linear unbiased predictor. Its explicit system isThe sign of the last term follows from the displayed convention $\Sigma w+\eta 1=c$. To derive the system, differentiate $Q(w)+2\eta (1^{T}w-1)$. To prove the minimum, let $w_{\ast }$ solve it and let $w=w_{\ast }+d$ be another feasible choice. Then $1^{T}d=0$, soA positive-definite matrix $\Sigma$ makes equality possible only for $d=0$. This argument uses second moments, not a normal distribution assumption. Ordinary kriging is the special case of universal kriging with $X=1$ and $x_0=1$.

For an original one-dimensional teaching example, observe $z=(1,3,2{)}^T$ at $s=(0,1,3)$, and predict at $s_0=2$. Choose unit marginal variance, exact observations without measurement noise, an unknown constant mean, and the exponential covariance functionThe exponential covariance function corresponds to a stationary Ornstein-Uhlenbeck process, so it gives a valid covariance model. Its semivariogram is $1-2^{-|h|}$. The resulting matrix and vector areSolving the displayed system givesThese values can be checked directly: the weights sum to one and $\Sigma w=(41/116,35/58,35/58{)}^T=c+(3/29)1$. Substitution into $Q(w)$ gives $18/29$. Nearest-neighbor interpolation must choose between the two equally close samples at one and three, returning either three or two; their error variances under this model are both $1+1-2(1/2)=1$. Thus the benefit is quantified under the chosen model, without knowing the unobserved target value.

The upper panel applies the same ordinary kriging calculation at each target location. The lower panel shows the square root of its error variance, in the units of the observations. It is zero at the exact observation locations and rises between and beyond them. No confidence interval is inferred from a variance alone; a probability distribution assumption is needed to attach a coverage level.

The weights are not required to be positive. Keep the same observations and target, but replace the covariogram by the smoother Gaussian kernel $C(h)=2^{-h^2}$. ThenThe negative coefficient compensates for dependence between samples; the constraint eliminating bias still holds. Kriging is not generally a convex combination, and its prediction need not stay between the observed values. A smaller model-based error variance does not establish that one covariance function is better supported by these three observations.

Set $v_{\infty }={\sigma }^2/(2\theta )$ and draw $X_0\sim \mathcal{N}(\mu ,v_{\infty })$ independently of future Brownian increments. The Ornstein-Uhlenbeck process is then a strictly stationary process, withEvery time has the same normal distribution, and the covariance function depends only on the time lag. For a Gaussian process these properties also make all its finite-dimensional distributions invariant under a common time shift. The stationary distribution balances the continual random forcing against the contracting drift.^[ref]

With a fixed $X_0=x_0$, the covariance function instead isIn particular, it is zero when $s=0$, since the initial value has no uncertainty. The second term records the fixed start; it is the general scaling of the zero-start Ornstein-Uhlenbeck covariance. The pure exponential covariance function displayed above assumes stationary initialization. For a fixed start, that function emerges as the late-time limit at fixed lag, while the one-time probability distributions approach $\mathcal{N}(\mu ,v_{\infty })$ by convergence in distribution.

For the stipulated $\theta =1$, $\mu =2$, $\sigma =\sqrt{2}$, stationary initialization has variance one from the outset. Its autocorrelation is $e^{-|h|}$, so the correlation time is one: a lag of one reduces the autocorrelation to $e^{-1}\approx 0.3679$. Its raw second moment $\mathbb{E}[X_s{X}_t]=4+e^{-|t-s|}$ includes the squared expected value; it must not be confused with the covariance.

The upper panel shows three simulated sequences with independent stationary initial values and exact transitions at spacing $0.05$. Joining their points only guides the eye; it does not simulate a continuous sample path between those points. The lower panel shows analytic covariance functions: the stationary curve and, for comparison, $\mathrm{Cov}(X_s,X_{s+h})$ for fixed initialization at two different ages $s$. These are simulated samples and model curves, not measured trajectories or empirical covariance estimates.

Condition throughout on the training inputs and $x_{\ast }$. Interpret $\theta$ here as the fixed covariance function and noise hyperparameters. Assume observation errors ${ϵ}_i$ are independent standard normal random variables, independent of the Gaussian process $f$, and writeThe independent-error assumption is the standard Gaussian process regression interpretation of the observation model; the single-observation marginal noise law alone would not specify cross-observation noise correlations. We take ${\sigma }^2>0$ so that $C$ is a positive-definite matrix. More generally the inverse formulas below apply whenever the indicated noisy covariance matrices are invertible.

The Gaussian process prior makes $(f_{\ast },\mathbf{f})$ a Gaussian random vector. Adding the independent Gaussian noise vector to $\mathbf{f}$ preserves a multivariate normal distribution. Its mean is zero, its observation covariance matrix is $C$, and $\mathrm{Cov}(f_{\ast },y_i)=k(x_{\ast },x_i)$ because the noise is independent of $f_{\ast }$. ThereforeA degenerate latent variance is allowed; the displayed multivariate normal distribution then denotes a Gaussian law even if it has no full-dimensional probability density.

Additional observations weakly decrease the posterior variance at every fixed prediction input. Apply part (ii) repeatedly to add observations one at a time. An observation whose remaining conditional covariance with $f_{\ast }$ is nonzero strictly decreases that posterior variance; one with zero remaining conditional covariance leaves it unchanged. Large observation noise reduces the information gained through the denominator $v_{n-1}(x_n)+{\sigma }^2$.

For fixed Gaussian process hyperparameters and a Gaussian likelihood, the posterior variance depends on the observation locations and noise level, but not on the observed numerical outputs. Hence the reduction holds for each possible new output, rather than only on average. The predictive variance of a future noisy observation also decreases by the same amount because its extra independent noise variance is fixed. Refitting the hyperparameters after adding data can change the covariance function; the monotonicity statement compares conditioning under one fixed model.