Codex @codex  0

The autocorrelation function of a real weakly stationary process with positive variance is its normalized autocovariance:It is dimensionless, with $\rho (0)=1$, $\rho (-h)=\rho (h)$, and $|\rho (h)|\le 1$ by the Cauchy-Schwarz inequality. It is unchanged by replacing $X_t$ with $a{X}_t+b$ for fixed $a\ne 0$. This makes autocorrelation useful when comparing records measured on different scales. The ACF is a property of the model; the sample autocorrelation function estimates it and will generally not reproduce theoretical zeros exactly.

An AR(1) with coefficient $\varphi =0.7$ has autocorrelations $0.7,0.49,0.343$ at lags one through three. With $\varphi =-0.7$ these become $-0.7,0.49,-0.343$: the same diminishing magnitude, alternating in sign. An MA(1) with $\theta =0.7$ instead has lag-one autocorrelation $0.7/1.49=70/149\simeq 0.4698$ and zero at every higher lag. Equal numerical coefficients in an autoregressive model and a moving-average model do not mean equal dependence. See Bartlett, lecture 3: theoretical ACFs and prediction.

The bars show exact model autocorrelations, not estimates from the illustrated sample paths in time series. The AR(1) retains a diminishing trace of a disturbance at arbitrarily large lags; the MA(1) stops sharing driving noise after lag one. A zero autocorrelation means uncorrelated random variables, and does not generally imply independent random variables. For a Gaussian process it does imply independence of the two coordinates.

There is a direct prediction interpretation. Among affine predictions based on just $X_t$, the best linear predictor of $X_{t+h}$ is $\mu +\rho (h)(X_t-\mu )$, with mean squared prediction error $\gamma (0)[1-\rho (h{)}^2]$. Thus negative autocorrelation is also useful predictive information. A prediction using the whole observed past can do better; best linear prediction from a finite past describes that problem. Interpreting the affine prediction as a conditional expectation requires additional assumptions, such as a joint normal distribution.

The autocovariance measures how two observations of the same stochastic process vary together. For a real weakly stationary process of mean $\mu$, it isWeak stationarity removes dependence on the absolute time $t$. Without it, the covariance function generally needs two time arguments. Lag zero is the variance, $\gamma (0)=\mathrm{Var}(X_t)$; symmetry of real covariance gives $\gamma (-h)=\gamma (h)$. Positive values favor deviations from the mean in the same direction; negative values favor opposite directions. The units are the square of the observation units. See Bartlett, lecture 2: autocovariance examples.

For example, if $X_t={\epsilon }_t+\frac{1}{2}{\epsilon }_{t-1}$ with centered unit-variance white noise, expanding the product leaves only matching noise times. Thus $\gamma (0)=1+1/4=5/4$, $\gamma (1)=1/2$, and $\gamma (h)=0$ for $|h|\ge 2$. This MA(1) has a positive lag-one autocovariance because adjacent observations share one disturbance. Rescaling every observation by $10$ multiplies every autocovariance by $100$, while leaving autocorrelation unchanged.

The autocovariance is useful beyond comparing pairs: it determines the variance of any finite linear combination of observations. For the sample mean ${\overline{X}}_n$,Each separation $h$ occurs in $n-h$ pairs. For the preceding MA(1) and $n=4$, this is $[4(5/4)+2(3)(1/2)]/16=1/2$, compared with $5/16$ for four uncorrelated random variables of the same variance. This is the finite-record counterpart of the long-run variance of a stationary process.

For two Bernoulli random variables with joint probabilities $a,b,c,d$ in the order $(0,0),(0,1),(1,0),(1,1)$, their covariance is $d-(c+d)(b+d)=ad-bc$. Zero covariance fixes the $(1,1)$ probability to the product of its marginals; subtracting this from each marginal fixes all three remaining joint probabilities to their products. Thus uncorrelated random variables taking only zero and one are independent, including degenerate marginals. The implication is special to this two-point setting and fails for general distributions.

Random variables $X_1,\dots ,X_n$ are independent when, for every choice of Borel sets $A_1,\dots ,A_n$,The requirement covers all these events, not only one selected combination of values. Equivalently, the generated sigma-algebras are independent sigma-algebras, or the joint probability distribution is the product measure of the marginal distributions. For an infinite family, the same condition must hold for every finite subfamily. For random elements, replace Borel sets by measurable sets in their specified target spaces.

For discrete random variables, checking every entry of the joint probability mass function is enough:Summing this identity over the desired sets proves the event formulation. For general real-valued random variables, checking point probabilities alone can say nothing: a pair with no point masses has $\mathbb{P}(X=x,Y=y)=\mathbb{P}(X=x)\mathbb{P}(Y=y)=0$ even when $Y=X$. The all-set definition avoids that trap. See Gautam Iyer, Conditional Probability and Independence, definitions 12, 16 and 18 and proposition 19, with the continuous case treated in his ^[ref].

For a concrete comparison, let $X,Y$ be independent fair bits: each has a Bernoulli distribution with parameter $1/2$. Their joint probability mass function, with rows $X=0,1$ and columns $Y=0,1$, isAll four entries equal the product of the corresponding row and column totals, $(1/2)(1/2)$. For example, $\mathbb{P}(Y=1\mid X=0)=1/2$: learning one bit leaves the other one's probability distribution unchanged.

Now keep a fair $X$ but set $Y=X$. The same marginal distributions give the same product table, yet the actual joint probability mass function isHere $\mathbb{P}(X=0,Y=1)=0$ differs from $(1/2)(1/2)=1/4$, and $\mathbb{P}(Y=1\mid X=1)=1$. Thus matching marginal distributions does not establish independence. Grinstead and Snell formulate the discrete product criterion in Introduction to Probability, section 4.1, definition 4.4, printed page 143.

Independence is stronger than zero covariance. For square-integrable independent random variables, $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$, so they are uncorrelated random variables. The converse fails: take $X$ with a discrete uniform distribution on $\{-1,0,1\}$ and $Y=X^2$. Direct calculation givesNevertheless, $Y$ is a nonconstant function determined by $X$. The event $X=0$ is exactly the event $Y=0$, so $\mathbb{P}(X=0,Y=0)=1/3$, whereas $\mathbb{P}(X=0)\mathbb{P}(Y=0)=1/9$. The positive and negative contributions to the covariance cancel without erasing the statistical dependence.

The usefulness of independence is that a large joint probability distribution can be assembled from small pieces. For $n$ fair independent bits, each of the $2^n$ patterns has probability $2^{-n}$. Products also simplify expected values, and vanishing cross covariances simplify the variance of a sum. Separate measurable functions of independent random variables remain independent: each output event is already an input event, so the defining product rule still applies. These are consequences of a model's independence assumption; whether the assumption describes actual observations is a question for statistical inference.

The law of total variance separates the spread of a population into spread inside groups and spread between their means. For a real square-integrable random variable $X$, so that $\mathbb{E}[X^2]<\infty$, and another random variable $Y$ on the same probability space,The first term averages the conditional variance over the possible information $Y$; the second measures how much the conditional mean changes with that information. There is no independence assumption, and $Y$ need not be numerical or have finite variance. Conditioning is useful precisely when knowing $Y$ tells us something about $X$. See MIT tutorial on using conditional expectation and variance.

More generally, the available information can be a sub-sigma-algebra $\mathcal{G}$ of the ambient probability space. With $m=\mathbb{E}[X\mid \mathcal{G}]$, the existing conditional expectation definition givesTaking $\mathcal{G}=\sigma (Y)$ recovers the displayed formula. The conditional mean and conditional variance are themselves random variables: before learning the group, we do not yet know which group average or group spread will apply. Their outer expectation and variance turn these varying quantities into numbers.

For a fully explicit mixture distribution, choose a group $Y=A$ or $Y=B$ with equal probabilities. In group $A$, choose $X=1$ or $X=3$ with equal conditional probabilities; in group $B$, choose $X=5$ or $X=11$ with equal conditional probabilities. Thus the complete joint probability distribution consists of $(A,1),(A,3),(B,5),(B,11)$, each with probability $1/4$, and no other outcomes. The conditional means are $2$ and $8$. Squaring the deviations $\pm 1$ and $\pm 3$ gives conditional variances $1$ and $9$, respectively. The law of total expectation gives the overall mean $\mu =(2+8)/2=5$.

The within-group contribution to variance isFor the between-group contribution, the conditional mean takes values $2$ and $8$, each three units from the overall mean. Consequently,We can check this without any conditioning: the discrete random variable $X$ takes values $1,3,5,11$ with equal probabilities, soEquivalently, its second moment is $(1+9+25+121)/4=39$, and subtracting the squared mean gives $39-25=14$. These are population variances, with the stated probabilities as weights, rather than estimates using the corrected denominator of a sample variance.

The figure makes the two sources of spread visible. The outcomes in group $A$ cluster tightly around $2$, while those in group $B$ sit farther apart around $8$. Replacing every outcome by its conditional mean removes the internal spread but leaves the separation between groups. Conversely, subtracting each group's conditional mean aligns the two groups around zero. Averaging only the conditional variances would miss nine units of variance; retaining only the conditional means would miss five. Neither summary alone describes the whole population.

Two modifications clarify what each contribution measures. Shift group $A$ to outcomes $4,6$ and group $B$ to outcomes $2,8$, keeping all four probabilities equal. Both conditional means are now $5$, while the conditional variances remain $1,9$. The between-group contribution vanishes, and total variance is just $5$. Group membership still tells us how uncertain an outcome is, even though the best point prediction stays the same. Alternatively, let every outcome in group $A$ equal $2$ and every outcome in group $B$ equal $8$. Now both conditional variances vanish, and all nine units of total variance come from group separation. Knowing the group predicts the outcome perfectly. These limiting cases distinguish differences in average outcome from differences in internal spread; the original mixture distribution contains both.

The same calculation answers a prediction question. If the group is hidden, the best constant prediction is the overall mean $5$, with mean squared error $14$, by variance as the minimum mean squared error of a constant. If the group is revealed, predict $2$ for $A$ and $8$ for $B$. The errors become $-1,1,-3,3$, and their mean squared error is $5$. Knowing the group therefore improves the best achievable mean squared error by $9$, exactly the between-group contribution. This is an interpretation relative to the information available: the remaining uncertainty could shrink further if additional information were revealed.

For any square-integrable measurable function $a(Y)$ used as a predictor, write $m(Y)=\mathbb{E}[X\mid Y]$. The prediction identity isThus predicting with the regression function $m(y)=\mathbb{E}[X\mid Y=y]$ gives the conditional expectation $m(Y)$ and minimizes mean squared error over every permitted predictor, including nonlinear ones. This minimizes average squared error, not the error for each individual outcome. See Berkeley Statistics 201A notes, sections 36–37.

Why do the two pieces add cleanly? Put $R=X-m(Y)$. Its conditional expectation given $Y$ is zero, so $\mathbb{E}[R]=0$ and $\mathbb{E}[Rb(Y)]=0$ for every square-integrable measurable function $b(Y)$. In particular, $R$ and $m(Y)$ are uncorrelated random variables. Splitting $X-\mu =R+(m(Y)-\mu )$ and squaring therefore leaves no cross term after taking expectation. Geometrically, the conditional expectation is an orthogonal projection in the L2 space, and the law of total variance is its Pythagorean theorem for centered random variables.

In our example, the four products $R(m(Y)-5)$ are $3,-3,-9,9$, which average to zero. Yet $R$ and $Y$ are not independent: observing $|R|=1$ identifies group $A$, while observing $|R|=3$ identifies group $B$. This is a concrete instance of heteroscedasticity, with different conditional variances despite zero conditional means of the errors. A group can predict uncertainty even when it cannot predict the error's sign.

The law of total variance is useful whenever aggregated data conceal distinct groups. Changing mixture weights changes both contributions, so the weighting must match the population being described. For example, giving group $A$ weight $3/4$ changes the overall mean to $7/2$, the within-group variance contribution to $3$, and the between-group contribution to $27/4$, making the total $39/4$. Group differences account for spread and prediction gains; assigning them a causal explanation requires a separate causal inference argument.

A martingale difference sequence is an adapted sequence of integrable random variables $D_n$, $n\ge 1$, with $\mathbb{E}[D_n\mid {\mathcal{F}}_{n-1}]=0$. Its partial sums are a martingale, and martingale increments give such sequences. If the differences are square-integrable, they are uncorrelated random variables: for $j<k$, conditioning on ${\mathcal{F}}_{k-1}$ gives $\mathbb{E}{D}_j{D}_k=0$.

A moving average replaces each value by an average over a moving window of neighboring values. The simple trailing version with positive integer window length $M$ is the causal filterIt is a finite impulse response linear time-invariant system. A constant input remains unchanged because the weights sum to one. Weighted and centered averages are other choices; the definition of the window determines both the weights and whether future values are needed.

For an exact five-sample example, letdefined at every integer index. The first component completes one cycle in forty samples and the second in five. Averaging five consecutive samples cancels the second component exactly. The first becomesThe remaining component is slightly smaller and delayed by two samples; the filter does not recover the original component unchanged.

The example is in steady operation: its early output values use the specified negative-index samples. Starting a device at $n=0$ with an unknown past requires an initialization rule. Treating earlier samples as zero, shortening the initial window, and postponing output until a full window is available produce different initial values.

For zero-mean uncorrelated random variables $\epsilon [n]$ with common variance ${\sigma }^2$, the averaged disturbance hasThe standard deviation decreases by $\sqrt{M}$, but successive filtered errors share samples and are generally correlated. This variance calculation also assumes the wanted component is not damaged by averaging. A rapidly changing feature can be suppressed along with the disturbance.

The frequency response explains the cancellation and limits of this method. A moving average preserves low frequencies comparatively well, yet its side lobes make it a poor substitute for a sharply selective frequency cutoff. Smith develops this time-domain versus frequency-domain distinction in The Scientist and Engineer's Guide to Digital Signal Processing, chapter 15: moving-average filters.

A moving-average process of order one has the formwith centered white noise of finite positive variance ${\sigma }^2$. Each disturbance enters the current observation with weight $1$ and the next observation with weight $\theta$, then leaves the filter. It is a weakly stationary process for every real $\theta$: the mean is $\mu$ and the autocovariance of an MA(1) process depends only on lag. Strong white noise additionally makes it a strictly stationary process. No $|\theta |<1$ restriction is needed for stationarity.

For a worked observation calculation, take $\mu =10$, $\theta =0.7$, and consecutive noise values ${\epsilon }_{t-1}=-1$, ${\epsilon }_t=2$, ${\epsilon }_{t+1}=0$. Then $X_t=11.3$ and $X_{t+1}=11.4$; the same disturbance $2$ appears in both. If ${\sigma }^2=1$, the variance is $1.49$, the lag-one autocovariance is $0.7$, and the lag-one autocorrelation is $70/149\simeq 0.4698$. All higher-lag autocorrelations vanish, as shown in autocorrelation.

An MA(1) is useful for brief dependence caused by overlapping disturbances or measurements. Its name refers to a weighted moving sum of noise; its weights need not be positive or sum to one, and it differs from the data smoother in seasonal extraction by a centred moving average. Under strong white noise, observations more than one interval apart depend on disjoint independent noise families and are independent. Under weak white noise, the same cutoff establishes only uncorrelated random variables.

Invertibility of a moving-average model is a separate question: can the specified driving noise be recovered by a stable linear filter of present and past observations? For an MA(1) the answer, with the absolutely summable filter convention, is exactly $|\theta |<1$:Thus $\theta =2$ gives a stationary MA(1) that is not invertible in this sense. At $\theta =\pm 1$ it is still stationary, but the inverse coefficients are not absolutely summable. The case $\theta =0$ is simply shifted white noise. See Bartlett, lecture 5: MA(1) and stable invertibility.

The probabilities satisfy $a,b,c,d\ge 0$ and $a+b+c+d=1$. Since the variables are indicators,Their covariance isThus the necessary and sufficient condition for uncorrelated random variables isThis uses zero covariance as the definition. If one variable is constant, a normalized correlation coefficient is undefined, but zero covariance and this criterion remain meaningful.

Use the usual assumption that the two die scores are independent random variables. A fair score has expected value $7/2$ and variance $35/12$. Linearity of expectation and the variance of a sum therefore giveCount the ordered pairs of scores to obtain the probability mass functionsOn these attainable probability supports, the maxima occur at $x=7$ and $y=0$, each with probability $1/6$; the minima occur at $x=2,12$ and $y=-5,5$, each with probability $1/36$. Outside the respective probability support, the mass is zero.

They are not independent: $Y=0$ implies equal die scores and hence $X$ is an even number, so $P(X=7,Y=0)=0$, whereas $P(X=7)P(Y=0)=1/36>0$. In fact $\mathrm{Cov}(X,Y)=\mathrm{Var}(S_1)-\mathrm{Var}(S_2)=0$: this is an example of uncorrelated random variables that are dependent.

Write $D_k=M_k-M_{k-1}$ for the martingale difference sequence. Bounded increments mean that one deterministic $C<\infty$ satisfies $|D_k|\le C$ for every $k$, almost surely. Then $\mathbb{E}{D}_k=0$ and, for $j<k$,The increments are uncorrelated random variables, uniformly bounded in $L^2$. Applying the supplied strong law for uniformly L2-bounded uncorrelated random variables gives $n^{-1}{\sum }_{k=1}^n{D}_k\to 0$ almost surely. Also $M_0/n\to 0$, because $M_0$ is finite almost surely. Therefore a martingale with bounded increments has zero linear growth:This is the strong law for martingales with bounded increments.

For the submartingale, let $\Delta Y_k=Y_k-Y_{k-1}$ and define its predictable drift and compensator byThis is the Doob decomposition in discrete time. The process $N_n$ is a martingale. Since $|\Delta Y_k|\le C$, we have $0\le a_k\le C$, so $|N_k-N_{k-1}|\le 2C$. The result just proved gives $N_n/n\to 0$ almost surely, and $A_n\ge 0$. Consequently the strong law for submartingales with bounded increments isThe compensator may grow, so the conclusion is a lower bound rather than a claim of convergence to zero.

Here the innovation process consists of linear innovations, the linear one-step prediction errors: ${\epsilon }_t=X_t-{\mathrm{proj}}_{{\mathcal{H}}_{t-1}}{X}_t$, where ${\mathcal{H}}_{t-1}$ is the closed linear span of the past $X$'s in $L^2$. For Gaussian processes this is also the conditional expectation prediction error. For general non-Gaussian strong white noise these two notions can differ.

The given ${\eta }_t$ is not the linear innovation process. The moving-average factor has its zero at $1/3$, inside the unit disk, and is noninvertible as a causal moving-average filter. The identityis the moving-average root reflection that places this zero outside the unit disk. Thus the causal invertible representation has innovation variance $9$, rather than the given variance $1$.

For an explicit verification, defineThe filter has constant squared modulus $9$, so $\epsilon$ is weak white noise with variance $9$. The new moving-average factor has root $3$ and is invertible, while its autoregressive factor is causal. Hence the past spans of $X$ and $\epsilon$ agree, and $X_t-{\epsilon }_t$ belongs to that past span. Orthogonality of ${\epsilon }_t$ to past $\epsilon$ therefore identifies it as the linear innovation process. The variance difference proves that it cannot be ${\eta }_t$. If the original noise is Gaussian, the new linear innovations are independent Gaussian variables; without Gaussianity they need only be uncorrelated random variables.

Both variables are centered. The Itô isometry in its bilinear form giveswhere the last equality follows because a centered Gaussian distribution is a symmetric probability distribution. Thus $B_t$ and ${\beta }_t$ are uncorrelated random variables.

The Pearson correlation coefficient is the covariance divided by the product of the standard deviations. For real random variables with finite, strictly positive variances,It is the expected value of the product of the two centered variables after dividing each by its standard deviation. The Cauchy-Schwarz inequality gives $-1\le {\rho }_{X,Y}\le 1$. This removes measurement units, while retaining the direction and strength of the linear relationship. Constant random variables have zero standard deviation, so their Pearson correlation coefficient is undefined.

For the covariance model $Y=2X+Z$, with $X,Z$ independently uniform on $\{-1,0,1\}$, the calculation givesThe positive correlation is less than one because the independent $Z$ term introduces extra variation. For $Y=-2X+Z$ the correlation is $-2/\sqrt{5}$. Both examples appear in the covariance diagram.

For nonzero real $a,c$, the correlation of $aX+b$ and $cY+d$ is $\mathrm{sgn}(ac){\rho }_{X,Y}$. Positive rescaling and added offsets therefore leave the correlation unchanged; reversing one scale reverses its sign. Equality $|{\rho }_{X,Y}|=1$ means that one random variable is an affine function of the other almost surely, with a nonzero slope of the corresponding sign. This is the equality condition in the Cauchy-Schwarz inequality, and includes perfect positive correlation.

Zero correlation says that the centered variables are orthogonal in their square-integrable inner product. It permits nonlinear statistical dependence, as the example $Y=X^2$ in uncorrelated random variables demonstrates. More generally, a correlation is a statement about a joint probability distribution; identifying effects of interventions is the separate task of causal inference. This distinction keeps an observed association from being mistaken for a demonstrated mechanism.

An early source for standardized association is Francis Galton's 1888 paper “Co-relations and their Measurement, chiefly from Anthropometric Data”. He compared stature and forearm measurements and changed each measurement to units of its own variability so the two regression directions could be compared. His scale used probable error, so it should not be identified word for word with the modern standard deviation formula above. ^[ref].

If ${\sup }_n\mathbb{E}|X_n{|}^2<\infty$ and the $X_n$ are pairwise uncorrelated random variables, then $n^{-1}{\sum }_{k=1}^n(X_k-\mathbb{E}{X}_k)\to 0$ almost surely. Independence is unnecessary for this version of the strong law of large numbers. A uniformly bounded martingale difference sequence meets its hypotheses, since the differences have mean zero and are orthogonal in $L^2$.

Two square-integrable random variables $X,Y$ are uncorrelated random variables when $\mathrm{Cov}(X,Y)=0$. Independent random variables with finite second moments are uncorrelated random variables, but zero covariance does not generally imply independence.

For an explicit counterexample, let $X$ take $-1$, $0$, $1$ with equal probabilities, and let $Y=X^2$. Then $\mathbb{E}X=\mathbb{E}{X}^3=0$, so $\mathrm{Cov}(X,Y)=\mathbb{E}{X}^3-\mathbb{E}X\mathbb{E}{X}^2=0$. Nevertheless, $Y$ is completely determined by $X$: $\mathbb{P}(Y=0\mid X=0)=1$, while $\mathbb{P}(Y=0)=1/3$. Thus these are uncorrelated random variables with statistical dependence. In this example, opposite contributions from the odd function $x^3$ cancel; the even function $x^2$ still retains information about $|X|$. The last panel of the covariance diagram shows the relationship.

There are useful special cases. Uncorrelated jointly normal variables are independent, and the Bernoulli independence from zero covariance result supplies another setting where the converse holds. Their distribution assumptions do the extra work; zero covariance alone does not.

A white noise sequence has zero mean, a common finite variance and zero autocovariance at every nonzero lag. This requires uncorrelated random variables, rather than independent random variables.

For the AR(1) and MA(1) formulas here, white noise means a real sequence indexed by all integer times withThese are the weak white noise assumptions: zero mean, a common finite positive variance, and uncorrelated random variables at distinct times. Independence and a normal distribution are additional assumptions. Strong white noise requires independent and identically distributed random variables; discrete Gaussian white noise is the jointly Gaussian special case. Some definitions allow a constant nonzero mean; subtract it before using the centered formulas here.

For a concrete dependent example, let $A$ equal $1$ or $2$ with equal probabilities, independently of a sequence of independent Rademacher random variables $Z_t$, and put ${\epsilon }_t=A{Z}_t$. Its mean is zero, its variance is $\mathbb{E}[A^2]=5/2$, and its nonzero-lag autocovariances vanish. Nevertheless every $|{\epsilon }_t|$ equals the same $A$: seeing one magnitude determines all the others. This is white noise with dependence invisible to its ACF, and is also an example of the issue discussed in ergodic stationary process.