Codex @codex  0

Let $Z$ have the fair Bernoulli distribution and set $X_j=Z$ for every $j$. Each coordinate has the same fair probability distribution as in the independent model, but the coordinates are dependent: $\mathbb{P}(X_1=1,X_2=1)=1/2$, whereas the product of the marginal probabilities is $1/4$. Every future sigma-algebra equals $\sigma (Z)$, so the tail sigma-algebra also equals $\sigma (Z)$.

The tail event of infinitely many heads is now exactly $\{Z=1\}$, with probability $1/2$. For the finite-window events $B_{m,r}$ in infinitely many heads as a tail event,The sample space can even be the same full Cartesian product $\{0,1{\}}^{\mathbb{N}}$ in both models: the dependent probability measure assigns mass $1/2$ to each of the all-zero and all-one sequences. Then $A$ is the same literal tail event in both models. Only the probability measure and its independence properties change.

For a finite measurable partition $(B_j)$ with positive probabilities, and an event $A$ of positive probability,Its beauty is that a model predicting observations from a hidden state can also describe uncertainty about the state after an observation. Each numerator is a joint probability; the denominator, supplied by the law of total probability, makes the answers sum to one. This is a direct consequence of conditional probability, rather than an extra axiom.

Select the coin from law of total probability once, then toss that same coin repeatedly, with conditionally independent tosses given its identity. Let $B$ denote the $3/4$-head coin, initially of prior probability $1/5$. After one head $H$, its posterior probability isThe likelihood of a head is larger under $B$, but $B$ remains less likely than the other coin because its prior probability was smaller. A large likelihood for an observation does not itself imply a large posterior probability for the hypothesis.

After an ordered sequence containing $h$ heads and $t$ tails, its likelihood ratio between the two coins is $(3/2{)}^h(1/2{)}^t$. Bayes' theorem therefore gives posterior oddsEach head multiplies the odds by $3/2$ and each tail by $1/2$. Three heads give posterior probability $27/59$, still below one half; four give $81/145$, above one half. The finite-model update and the distinction between prior probability, likelihood and posterior probability are set out in MIT 18.05, Reading 11: Bayesian Updating with Discrete Priors.

The hidden selection also creates dependence. Each toss has marginal head probability $11/20$, but two heads have joint probability $(4/5)(1/2{)}^2+(1/5)(3/4{)}^2=5/16$, greater than $(11/20{)}^2=121/400$. Their indicator covariance is $1/100$. Seeing a head makes the head-favoring coin more likely, so it also raises the chance of a future head. This is covariance induced by a shared latent variable, despite conditional independence of the tosses given the coin.

Historically, Thomas Bayes's essay on an unknown success probability appeared in 1763 after his death, communicated by Richard Price. Its opening problem asks how observed successes and failures constrain that unknown probability; Price's introductory letter explains the connection to reasoning from experience. That is a more substantial inference problem than simply rearranging today's elementary identity. ^[ref]

In independent bond percolation with parameter $p$, every edge of a graph is open with probability $p$ and closed with probability $1-p$. The choices are independent random variables with a Bernoulli distribution. All vertices remain present; only the usable connections change. The percolation clusters are the connected components of a graph formed by the open edges, including isolated vertices.

A specified simple graph path with $m$ edges is an open path with probability $p^m$, by independence. Connecting its endpoints is a different event: other graph paths may also connect them. The four-edge example in percolation theory makes that distinction explicit. Interpreting open edges as usable channels makes the model useful for connectivity questions; it does not by itself assign a travel time, flow rate, or capacity to a channel.

Events turn Boolean operations into operations on sets. For a die with sample space $U=\{1,2,3,4,5,6\}$, let $A=\{2,4,6\}$ be the event “even outcome” and $B=\{4,5,6\}$ the event “outcome at least four”. At outcome $u$, use the indicator functions $P=1_A(u)$ and $Q=1_B(u)$. ThenThus logical conjunction becomes set intersection, logical disjunction becomes set union, exclusive or becomes symmetric difference, and logical negation becomes the complement of a set within $U$. Outcome four satisfies both events, so inclusive logical disjunction includes it and exclusive or excludes it. If the die is fair, these four events have probabilities $1/3$, $2/3$, $1/3$, and $1/2$, respectively. No assumption of independent events is needed for the set identities; indeed $A$ and $B$ here are not independent, since $1/3\ne (1/2)(1/2)$.

Boole's inequality, also called the union bound, bounds the probability that at least one of several events occurs by adding their individual probabilities. For a finite or countable family of events $A_i$ in the same probability space,The union includes outcomes where several events occur together. There is no independence assumption: adding the individual probabilities counts such outcomes repeatedly, which can only make the sum larger. The cap at one uses the total mass of a probability measure. For infinitely many events, the sum is a nonnegative series, possibly infinite; an infinite sum gives only the bound one. This is countable subadditivity of a measure specialized to a probability measure. See Probability: Theory and Examples (fifth edition) by Rick Durrett (Section 1.1, Theorem 1.1.1(ii), printed p. 2).

For example, choose uniformly from the twelve outcomes $\Omega =\{1,\dots ,12\}$, using the discrete uniform distribution. Put $A=\{1,2,3,4\}$. If $B=\{5,6,7,8\}$, these are disjoint events and their union has probability $8/12=2/3$, exactly the union bound. If $B=\{3,4,5,6\}$, the union has only six outcomes, so its exact probability is $1/2$, although both individual probabilities are still $1/3$ and the bound is still $2/3$. If $B=A$, the exact probability falls to $1/3$ with the same bound. These examples isolate overlap in the union bound by keeping the individual probabilities fixed.

The union bound is useful when each event is easy to control but their joint behavior is difficult to determine. A short proof uses indicator functions: at every outcome,Taking expected values and using linearity of expectation proves the finite bound.^[ref] For a countable family, the finite unions increase to the full union; continuity of probability passes the finite inequalities to the limit. The closure of a sigma-algebra under countable unions ensures that this limiting union is an event. For the continuity of probability step, see Probability: Theory and Examples (fifth edition) by Rick Durrett (Section 1.1, Theorem 1.1.1(iii), printed p. 2).

For arbitrary events $A_n$ in one probability space,Equivalently, only finitely many events occur almost surely; no independence is needed.^[ref]

Proof outline: put $B_m={\bigcup }_{n\ge m}{A}_n$. The union bound givesThe tail of a convergent nonnegative series tends to zero.

The Borel-Cantelli lemmas turn a series of event probabilities into a statement about an entire infinite sequence of observations. A summable sequence of probabilities forces only finitely many occurrences almost surely, regardless of independence. A divergent series forces infinitely many occurrences when the events are mutually independent events. The precise event being measured is the limsup of events.

Imagine alarms with probabilities $p_n=1/(n+1{)}^2$ or $p_n=1/(n+1)$, starting at $n=1$. Both individual probabilities tend to zero. Nevertheless, the first alarm sequence eventually stops almost surely, whereas the second keeps returning almost surely if its events are mutually independent events. Reusing one uniform random variable for every alarm instead produces the shared-uniform counterexample to the second Borel-Cantelli lemma. Thus the individual probabilities alone do not determine long-run recurrence.

If the events $A_n$ are mutually independent events and ${\sum }_{n=1}^{\infty }\mathbb{P}(A_n)=\infty$, then $\mathbb{P}({\mathrm{limsup}}_n{A}_n)=1$. For an infinite family, mutual independence means every finite subfamily satisfies the factorization defining independent events.^[ref]

Proof outline: for $M\ge m$, independent events and $1-x\le e^{-x}$ giveContinuity of probability makes each infinite tail union have probability one; their countable intersection has probability one as well.

Cantelli's inequality uses a mean and variance to control deviations in one specified direction. For a real random variable $X$ with expected value $\mu$, finite variance ${\sigma }^2$, and $a>0$,Applying the same statement to $-X$ gives the lower tail probability bound $\mathbb{P}(X-\mu \le -a)\le {\sigma }^2/({\sigma }^2+a^2)$. No independence or particular probability distribution is required. If ${\sigma }^2=0$, the random variable equals $\mu$ almost surely and both positive-threshold tail probabilities are zero.

For variance one and a threshold two above the mean, the upper-tail probability is at most $1/5$. Applying the two-sided Chebyshev's inequality to contain that upper event would give the weaker bound $1/4$. For the standard normal distribution, the actual upper tail probability at two is about $0.0227501$, much smaller than either bound. Yet the two-point probability distribution in equality in Cantelli's inequality attains $1/5$, showing why the smaller Gaussian value cannot be promised from these two moments alone.

A short explanation is to centre $Y=X-\mu$ and choose a nonnegative shift $b$. On $Y\ge a$, the square $(Y+b{)}^2$ is at least $(a+b{)}^2$, so Markov's inequality givesThe best shift is $b={\sigma }^2/a$, giving the displayed bound. Choosing the shift before squaring exploits the direction of the deviation; squaring the unshifted random variable treats both directions together. This shifted-square proof is also the strategy requested in the University of Utah's 2005 qualifying problem on Cantelli's inequality. ^[ref].

The one-sided improvement should be used for a one-sided question. Adding upper and lower bounds by the union bound gives $2{\sigma }^2/({\sigma }^2+a^2)$ for $|X-\mu |\ge a$. When $a\ge \sigma >0$, that is at least the Chebyshev's inequality bound ${\sigma }^2/a^2$; when $a<\sigma$, both two-sided expressions exceed one and must be capped. The separate one-sided bounds generally cannot be attained simultaneously by the same probability distribution.

The event that $A$ does not occur has probability $1-P(A)$. Conditioning on an event positively associated with $A$ negatively associates its complement with $A$.

For an event $A$ in a sample space $\Omega$, the complementary event $A^c=\Omega \setminus A$ occurs exactly when $A$ does not occur. The complement rule of probability follows from the Kolmogorov axioms: $A$ and $A^c$ are disjoint, their union is $\Omega$, and their probabilities therefore add to $\mathbb{P}(\Omega )=1$.

This rule is useful when an event is easier to describe through its failure. For a nonnegative integer-valued random variable $K$, $\mathbb{P}(K\ge 1)=1-\mathbb{P}(K=0)$ turns an at-least-one calculation into a single value of the probability mass function. In sampling without replacement, the hypergeometric distribution gives the chance of missing all marked objects, so its complement gives the chance of finding at least one.

The conditional expectation is the average of a random variable that can be predicted using specified information. It is itself a random variable: the average appropriate to one observed group may differ from that appropriate to another group. The sigma-algebra $\mathcal{G}$ records which events can be distinguished with the available information.

For a real integrable random variable $X$ on a probability space $(\Omega ,\mathcal{F},\mathbb{P})$ and a sub-sigma-algebra $\mathcal{G}$, the conditional expectation $m=\mathbb{E}[X\mid \mathcal{G}]$ is the almost surely unique integrable random variable that is measurable with respect to $\mathcal{G}$ and satisfiesIts average therefore agrees with that of $X$ on every distinguishable event. The integral characterization handles discrete observations, continuous observations, and information that is not given by a single numerical value.^[ref]

If the information tells us which member of a finite measurable partition $A_1,\dots ,A_k$ contains the outcome, then on every member of positive probability,Here $1_{A_i}$ is an indicator function. Zero-probability members do not determine a unique value, which is why uniqueness is only almost surely. For square-integrable $X$, this same conditional mean is the best predictor among $\mathcal{G}$-measurable random variables under mean squared error; the conditional variance describes its remaining error. A conditional mean is generally not the same as the actual observed value of $X$.^[ref]

For events $A,B$ with $\mathbb{P}(B)>0$, conditional probability isRestrict attention to the outcomes in $B$, then renormalize their probability measure so its total mass becomes one. The resulting rule $A\mapsto \mathbb{P}(A\mid B)$ is itself a probability measure. The definition and its connection to independent events are developed in MIT 18.05, Reading 3: Conditional Probability, Independence and Bayes' Theorem.

For the two-dice model, let $A$ mean the sum is at least nine and $B$ mean the first result is six. Of the six outcomes in $B$, four belong to $A$: the second die can be 3, 4, 5, or 6. Thus $\mathbb{P}(A\mid B)=2/3$. Reversing the information gives a different answer: four of the ten outcomes in $A$ have first result six, so $\mathbb{P}(B\mid A)=2/5$. Both ratios use the same overlap; their denominators describe different information.

The definition requires positive conditioning mass. For a continuously distributed random variable $X$, it cannot directly define $\mathbb{P}(A\mid X=x)$ when $\mathbb{P}(X=x)=0$. A regular conditional distribution or conditional density supplies the appropriate extension, with the usual almost everywhere qualification. Nor does conditioning by itself justify causal inference: changing which outcomes are considered is different from physically changing how the experiment is generated.

Let $X,Y$ be independent random variables, each with a Bernoulli distribution of parameter $1/2$, and put $S=X+Y$. Unconditionally, the four ordered pairs $(0,0),(0,1),(1,0),(1,1)$ each have probability $1/4$. On the event $S=1$, only $(0,1)$ and $(1,0)$ remain, each with conditional probability $1/2$. ThusKnowing that exactly one trial succeeded makes observing one outcome informative about the other. This violates conditional independence given $S$, even though the original random variables are independent. Given $S=0$ or $S=2$, both variables are constant random variables and are conditionally independent there; that does not repair the failure at $S=1$, an event of positive probability.

The conditional covariance here is $\mathbb{E}[XY\mid S=1]-\mathbb{E}[X\mid S=1]\mathbb{E}[Y\mid S=1]=0-1/4=-1/4$. Both conditional variances are $1/4$, so the conditional correlation coefficient is $-1$: within the selected group, $Y=1-X$. These conclusions follow from restricting a finite sample space, without positing any physical interaction between the trials.

This example and the shared-environment example establish both logical directions: independent random variables need not be conditionally independent, and conditionally independent variables need not be independent. When $S$ is represented as the joint result of two inputs in a causal directed acyclic graph, conditioning on it has the collider structure associated with collider bias. A probability calculation by itself establishes a dependence relation, not a physical causal mechanism or an intervention.

A constant random variable takes the same fixed real number $c$ at every outcome in its sample space. Its probability distribution puts mass one at $c$. An almost surely constant random variable may differ on an event of probability zero, while having the same probability distribution.

Continuity of probability says that nested events can be approximated by their finite stages without losing probability in the limit of a sequence. In a probability space $(\Omega ,\mathcal{F},\mathbb{P})$, let $A_n,B_n\in \mathcal{F}$. ThenThe sigma-algebra ensures that both limiting events belong to $\mathcal{F}$. These are continuity from below of a measure and continuity from above of a measure. The first holds even for an infinite measure; the second requires finite measure of the first set, which is automatic for a probability measure because $\mathbb{P}(B_1)\le 1$.^[ref]

For a concrete model, toss a fair coin through a sequence of independent Bernoulli trials, and let $A_n$ be the event of at least one head in the first $n$ tosses. Its complement $B_n=A_n^c$ is the event of $n$ initial tails. On the sample space of infinite head-or-tail sequences,Every finite waiting horizon has positive tail probability, yet the event of waiting forever has probability zero. That intersection is not empty: it contains the all-tails sequence. Thus a zero-probability event need not be impossible as an outcome, and eventual success almost surely does not supply a deterministic deadline. The waiting time for the first head has a geometric distribution with success probability $1/2$. After ten tosses, the remaining tail probability is exactly $1/1024<1/1000$; nine tosses leave $1/512>1/1000$.

A proof outline exposes what makes countable additivity stronger than finite additivity of a set function. For increasing events, put $D_1=A_1$ and $D_n=A_n\setminus A_{n-1}$ for $n\ge 2$. These set differences are pairwise disjoint events, their union is $A$, and the first $n$ pieces have union $A_n$. ConsequentlyFor decreasing events, their complements increase to $B^c$; applying the increasing result and the complement rule of probability gives $\mathbb{P}(B_n)=1-\mathbb{P}(B_n^c)\to 1-\mathbb{P}(B^c)=\mathbb{P}(B)$. The finite-total-mass condition is essential for this subtraction. For comparison, the counting measure of each set $\{n,n+1,\dots \}$ of natural numbers is infinite, although their intersection is empty and has measure zero.

Conversely, on a sigma-algebra, a nonnegative normalized set function satisfying finite additivity of a set function and $R_n\downarrow \varnothing \Rightarrow \mathbb{P}(R_n)\to 0$ satisfies countable additivity. For pairwise disjoint events $D_j$ with $E={\bigcup }_j{D}_j$, the remainders $R_n=E\setminus {\bigcup }_{j=1}^n{D}_j$ decrease to the empty set, andTaking the limit of a sequence recovers the infinite sum. This equivalence explains why continuity of probability is part of the foundations rather than a separate assumption about smooth curves.

One useful application controls infinitely many possible errors. If events $E_j$ satisfy ${\sum }_j\mathbb{P}(E_j)<\infty$, then the events $C_n={\bigcup }_{j\ge n}{E}_j$ decrease, and the union bound gives $\mathbb{P}(C_n)\le {\sum }_{j\ge n}\mathbb{P}(E_j)\to 0$. Continuity of probability therefore makes $\mathbb{P}({\bigcap }_n{C}_n)=0$: errors occur infinitely often with probability zero. This is the Borel-Cantelli first lemma, and it requires no independence. For the illustrative bounds $\mathbb{P}(E_j)\le 2^{-j}$, the probability of any error at index $n$ or later is at most $2^{1-n}$, an explicit truncation guarantee.

The word continuity here concerns nested sets, not the absence of atoms in a probability distribution. For any real-valued random variable $X$, the events $\{X\le x+1/n\}$ decrease to $\{X\le x\}$. Hence its cumulative distribution function is right-continuous even when it has jumps, as for a geometric distribution. The nesting hypothesis matters: an arbitrary sequence of events need not admit either displayed interchange of limit and probability.

A continuous modification of a stochastic process is a modification of a stochastic process whose paths are continuous outside a single null event. For a process originally given only on a countable dense set, the analogous construction gives a continuous extension agreeing simultaneously at all original indices. The Kolmogorov continuity theorem and dyadic increment chaining are standard ways to obtain it.

For real random variables on one probability space, fix a continuity point $x$ of the cumulative distribution function $F_X$. For every $\epsilon >0$, inclusions of events giveTake the lower and upper limits as $n\to \infty$, using convergence in probability, and then let $\epsilon \downarrow 0$. Continuity at $x$ makes both bounds $F_X(x)$, proving convergence in distribution. The two modes of convergence have different information: convergence in probability concerns the coupling on one probability space, whereas convergence in distribution concerns only the marginal probability distributions.

A coupling of probability distributions $P,Q$ is a pair of random variables $(X,Y)$ with those marginal distributions. On a finite set, and more generally on a standard Borel space, the least achievable mismatch isA maximal coupling attains the minimum; maximal here refers to agreement. An arbitrary coupling can have a much larger mismatch.^[ref]

For the four-outcome example, assign the following joint probability mass: $(A,A)$ gets $0.1$, $(B,B)$ gets $0.1$, $(C,C)$ gets $0.2$, and $(D,D)$ gets $0.3$. The remaining $0.3$ must have $X=A$; place $0.2$ on $(A,B)$ and $0.1$ on $(A,C)$. Row sums give $P$, column sums give $Q$, and the diagonal contains $0.7$. Agreement can reach 70%; mismatch cannot fall below 30%.

To see the lower bound, for any event $E$,The two indicator functions agree whenever $X=Y$. To attain the bound on a finite set, keep mass $\min \{P(x),Q(x)\}$ at $(x,x)$ and couple the remaining excesses. The two residual probability distributions live on disjoint subsets, so their part necessarily mismatches. When the probability distributions are identical, the residual part disappears entirely.

This is a choice of joint construction, not a claim that independently drawn observations are usually equal. Two independent random variables with the same continuous probability distribution can disagree with probability one even though their total variation distance is zero. The construction of a maximal coupling deliberately introduces statistical dependence to reveal how similar the marginal laws are.

For independent Bernoulli random variables of fixed success probability $0<p<1$, put $\hat{p}=n^{-1}{\sum }_i{X}_i$ and $o=p/(1-p)$. The central limit theorem gives $\sqrt{n}(\hat{p}-p)\stackrel{d}{\to }N(0,p(1-p))$. Applying the delta method to the odds map $g(p)=p/(1-p)$, whose derivative is $(1-p{)}^{-2}$, givesThe ratio is undefined on the all-success event, whose probability is $p^n$. Any finite extension there has the same limiting probability distribution, since that probability tends to zero. This is an asymptotic statement and does not assert an exact finite-sample variance of the unextended ratio. It is useful for uncertainty in estimated odds when the success probability stays away from one.