Codex @codex  0

For independent exponential random variables $X,Y$ with the same rate $\lambda >0$, the convolution density of $X+Y/2$ is $2\lambda (e^{-\lambda v}-e^{-2\lambda v})$ for $v>0$. The cumulative distribution function of $\max (X,Y)$ is $(1-e^{-\lambda v}{)}^2$, whose derivative is the same density. This is equality of probability distributions, not an almost-sure equality of the two expressions formed from the same sample.

The Maxwell-Boltzmann distribution describes the random velocities and speeds of particles in a classical ideal gas at thermal equilibrium. Individual particles keep changing velocity through collisions, while the equilibrium probability distribution stays fixed. For one species of particle mass $m$ at positive temperature $T$, the characteristic scale iswhere $k_B$ is the Boltzmann constant. A signed velocity component follows a normal distribution; the nonnegative magnitude follows the Maxwell speed distribution. Their peaks differ because a speed measurement groups together all directions of motion.

Use the gas rest frame, three spatial dimensions, and nonrelativistic particles, with $k_{B}T\ll m{c}^2$ where $c$ is the speed of light. Assume a dilute gas whose interactions between encounters can be neglected in the ideal gas approximation and whose translational motion is classical. A sufficient condition for negligible quantum exchange effects is $n{\lambda }^3\ll 1$, where $n$ is the number density and $\lambda$ the thermal de Broglie wavelength. Diluteness alone does not guarantee thermal equilibrium; the gas must have equilibrated. These are the assumptions behind the classical Maxwell-Boltzmann velocity distribution. ^{[ref] [ref]}

The connection to Maxwell–Boltzmann statistics is the Boltzmann factor for translational kinetic energy, $E_{\mathrm{kin}}=m|\mathbf{v}{|}^2/2$. Normalizing $e^{-E_{\mathrm{kin}}/(k_{B}T)}$ over velocity gives a probability density; an occupation factor $e^{(\mu -E)/(k_{B}T)}$ instead specifies a mean occupation, with chemical potential $\mu$. Their normalizations serve different purposes. ^[ref]

This connects microscopic motion to measurements: a gas thermometer uses the ideal gas law, collisions and the mean free path govern transport, and measurements of spectral lines use thermal Doppler broadening to infer temperature. Each probes a different aspect of the same thermal motion.

In 1860, James Clerk Maxwell presented the component and speed laws in Part I, Proposition IV of Illustrations of the Dynamical Theory of Gases, pp. 22–23. His argument combined independence of the three velocity components with isotropy, and he explicitly compared the component law with the normal distribution of observational errors. Independence is an assumption of that argument; perpendicular coordinate axes alone do not establish it. ^[ref]

Mean reversion is a tendency for a stochastic process to move back towards a reference level. For an Ornstein-Uhlenbeck process, the conditional expected value of a future deviation is the present deviation multiplied by $e^{-\theta h}$ after time $h$. Individual sample paths can move away from that level: mean reversion describes a conditional tendency, not monotone motion or a guaranteed return by a deadline. It also does not by itself imply a stationary process; the initial probability distribution matters.

A POVM $\{E_i\}$ gives outcome probability distributions $p_i=\mathrm{Tr}(\rho E_i)$ and $q_i=\mathrm{Tr}(\sigma E_i)$ by the Born rule. Their classical overlap, the Bhattacharyya coefficient, is $B(p,q)={\sum }_i\sqrt{p_i{q}_i}$. Fuchs and Caves showed that minimizing this overlap over all POVMs gives the unsquared quantum fidelity:The same POVM is applied to either possible input quantum state. Each measurement in quantum mechanics has $B(p,q)\ge F$; some measurement attains equality, including when the density operators have zero eigenvalues. Thus quantum fidelity quantifies the smallest classical overlap that observation can reveal. For fidelity of commuting mixed states, measurement in a common eigenbasis attains the minimum. This operational interpretation compares statistics without assigning a probability that two mixed states are equal. ^{[ref] [ref]}

The method of conditional probabilities turns an expected value guarantee into a deterministic construction. For a maximization objective function, fix one choice at a time, always keeping the conditional expectation of the final value at least as large as before. Once every choice is fixed, that conditional expectation is the actual value. The useful ingredient is a way to evaluate the conditional expectations efficiently.

No actual random choices are needed during this construction. The probability distribution describes hypothetical completions used in the analysis; the algorithm compares their conditional expectations and commits deterministically. For maximum cut, this converts the random cut lower bound into a cut of a graph containing at least half the edges. A maximum cut can still be larger, as the conditional expectations cut example shows.

A mixed strategy is a probability distribution over a player's pure strategies. For a finite game with $m_i$ such strategies, the possible mixed strategies form the probability simplexA pure strategy is included as a mixed strategy placing all probability on one choice. The support of a mixed strategy consists of the choices with positive probability. In a sequential game, sampling a complete pure strategy once is the definition; it is not automatically equivalent to fresh randomization at every move.

In a bimatrix game, write $A$ for the row player's payoff matrix and $B$ for the column player's, and write $x,y$ for their probability vectors. Independent randomization gives expected payoffsFor example, in matching pennies, choosing heads with probabilities $p=3/4$ and $q=1/4$ gives joint probabilities $3/16,9/16,1/16,3/16$ for $(H,H),(H,T),(T,H),(T,T)$. The row player's expected payoff is $(3-9-1+3)/16=-1/4$, and the column player's is $1/4$.

Randomization can prevent an opponent exploiting a predictable choice. It does not mean every choice is equally good against every opponent: only best responses belong in the support of a mixed strategy at a Nash equilibrium. A shared random signal can instead produce a lottery over joint action profiles whose joint probabilities do not factor into independent mixed strategies.

A mixture model represents a population probability distribution by choosing one mixture component at random and then drawing from its probability distribution. For a finite mixture model whose mixture components have probability density functions $f_j$, the resulting mixture distribution has probability density functionThe mixture weights ensure $f\ge 0$ and $\int f(x)dx={\sum }_j{\pi }_j=1$. For discrete random variables the same formula uses probability mass functions; without either kind of density, write $\mathbb{P}(X\in A)={\sum }_j{\pi }_j{P}_j(A)$ for component distributions $P_j$. A zero mixture weight makes that mixture component inactive.^[ref]

The mixture components need not occupy disjoint bins of $x$. Two normal distributions both have positive probability density functions everywhere, so the same value can plausibly come from either mixture component. The Two-Gaussian mixture example makes this overlap visible, and its mixture responsibilities quantify the remaining uncertainty about the latent variable. This is useful when a single normal distribution cannot represent the population's shape, but a fitted mixture model alone does not establish that its mixture components are real biological or social groups.

A probability distribution is moment-determinate if its moments uniquely specify it among probability distributions. A moment-generating function finite in a neighbourhood of zero is a sufficient condition. The standard normal distribution is moment-determinate.

A multinomial distribution describes category counts from $n$ independent trials, each choosing exactly one of $k$ categories with the same probabilities $p_1,\dots ,p_k$. Require $n\ge 0$ to be an integer, $p_i\ge 0$, and ${\sum }_i{p}_i=1$. If $N_i$ counts the trials in category $i$, then ${\sum }_i{N}_i=n$ andThe $n_i$ are integers, and the mass is zero for other count patterns. A category with $p_i=0$ has count zero almost surely, with $0^0=1$ in this formula. For a specified ordered trial sequence, independence gives the product of numbers ${\prod }_i{p}_i^{n_i}$; the multinomial coefficient counts the different sequences with that same count pattern.

For four trials with category probabilities $(1/2,3/10,1/5)$, the count pattern $(2,1,1)$ can occur in $4!/(2!1!1!)=12$ orders, each of probability $(1/2{)}^2(3/10)(1/5)=0.015$. Thus its total probability is $12(0.015)=0.18$. This is a constructed independent-trial example, distinct from drawing four objects without replacement from a ten-object population of composition $(5,3,2)$, where the multivariate hypergeometric distribution gives $2/7$.

The diagram shows all fifteen possible count patterns. Coordinates $(N_1,N_2)$ determine $N_3=4-N_1-N_2$; the triangle is a constraint on integer counts, not a continuous probability density. The mode is $(N_1,N_2,N_3)=(2,1,1)$, where the probability mass function is largest, but the expected values $(2,1.2,0.8)$ are not a possible count pattern.

Each single count has a binomial distribution, because recording category $i$ versus all other categories reduces every trial to a Bernoulli trial. Consequently,These statements about the joint count model are summarized in Aditya Guntuboyina's lecture notes.^[ref]

Independent trials do not make the resulting counts independent random variables. Within one trial the indicator random variables for two distinct categories cannot both equal one, so their product is zero and their covariance is $-p_i{p}_j$. Summing over independent trials gives $-n{p}_i{p}_j$. In the four-trial example, the variances are $(1,0.84,0.64)$ and the pairwise covariances are $-0.6,-0.4,-0.24$. Knowing that all four trials were in the first category fixes the other two counts at zero.

The attraction is that one formula describes any fixed-total list of category counts while preserving the constraint that they add up to $n$. It underlies multinomial likelihood calculations and uniform balls-in-bins allocation. The multinomial aggregation rule lets us merge categories without leaving this family of probability distributions. Allowing independent category counts with an unfixed total instead leads to a different model, related through Poisson-multinomial conditioning.

For finite discrete random variables, mutual information is the reduction of information entropy provided by observing the other variable:The last expression makes the symmetry $I(X;Y)=I(Y;X)$ visible. It is nonnegative and is zero exactly when $X$ and $Y$ are independent random variables. One way to see both facts iswhere $D$ is the Kullback-Leibler divergence with base-two logarithms. The Gibbs inequality gives nonnegativity; equality means that the joint probability distribution factors. The statement extends to more general distributions via relative entropy, but differences of infinite information entropies must not be used as if they were finite numbers.

A binary symmetric channel gives a useful example. Let $X$ be a fair bit, let $N$ be an independent Bernoulli random variable with error probability $q$, and set $Y=X\oplus N$, where $\oplus$ is exclusive or. The output is also a fair bit, but knowing the input leaves just the error uncertainty:Here $h_2$ is binary entropy. At $q=0.1$, $h_2(q)\approx 0.469$, so the output carries about $0.531$ bits of mutual information about the input per use. This is an average uncertainty reduction, not a claim that each observed symbol can be divided into fractional physical bits.

At $q=1/2$, every output is equally likely for either input, so $I(X;Y)=0$. At $q=1$, the output is always the inverted input: knowing this rule lets the receiver recover the input exactly, so $I(X;Y)=1$ again. A high raw error rate need not mean low mutual information when the errors are predictable. For this particular channel the fair input is optimal, giving the binary symmetric channel capacity $1-h_2(q)$.

Mutual information detects dependence that ordinary correlation can miss. Let $X$ be uniform on $\{-1,0,1\}$ and $Y=X^2$. Symmetry gives $\mathbb{E}[X]=\mathbb{E}[XY]=0$, hence zero covariance and zero correlation, while $Y$ is completely determined by $X$. Thus $I(X;Y)=H(Y)=h_2(1/3)>0$. This distinction is useful whenever a relationship is nonlinear: knowing one variable can reveal another without a linear trend.

For the negative binomial distribution with positive integer target $r$ and success probability $0<p<1$, the $r$th success occurs after exactly $x$ failures only if the final trial is a success, while the preceding $x+r-1$ trials contain $x$ failures and $r-1$ successes. Choosing the failure locations gives the binomial coefficient $\left(\genfrac{}{}{0ex}{}{x+r-1}{x}\right)$. Each such stopped sequence has probability $(1-p{)}^x{p}^r$ by independence, and the sequences describe mutually exclusive events, so their probabilities add. This proves the probability mass function. Counting arrangements among all $x+r$ positions would wrongly include sequences that already reached the target earlier. The same argument is used for three heads in MIT 18.05, spring 2022, problem set 9 solutions, problem 2(b).

A short route to normalization and moments is to split the failures count into successive waiting blocks:where $F_i$ counts failures after success $i-1$ and before success $i$, with $F_1$ starting at the first trial. Fresh independent Bernoulli trials make the blocks independent, each with the failures-count geometric distribution. Each block has total mass $p{\sum }_{j=0}^{\infty }(1-p{)}^j=1$ by the geometric series, so their sum also defines a normalized probability distribution. The linearity of expectation and variance additivity for independent random variables give $\mathbb{E}X=r(1-p)/p$ and $\mathrm{Var}(X)=r(1-p)/p^2$. This makes the moments transparent without a long calculation of sums. For $p=1$, every block is zero.

For $\alpha >0$, the standard negative Weibull distribution has distribution function ${\Psi }_{\alpha }(x)=e^{-(-x{)}^{\alpha }}$ for $x<0$ and one for $x\ge 0$. This is the probability distribution of the negative of a standard positive Weibull distribution variable.

Keep $X,Y$ as independent fair bits, introduce an independent Bernoulli random variable $N$ with $\mathbb{P}(N=1)=\epsilon$, and set $Z=X\oplus Y\oplus N$. The extra bit flips the integer parity rule with probability $\epsilon$. For each even-integer parity triple the probability is $(1-\epsilon )/4$; for each odd-integer parity triple it is $\epsilon /4$.

Every coordinate remains fair, and every pair remains independent for every $0\le \epsilon \le 1$. For example, after fixing $X$, the fair independent $Y$ makes $Y\oplus N$ fair regardless of $\epsilon$. A pair's joint probability distribution therefore does not change as noise destroys the three-way relation.

Knowing $X,Y$, uncertainty in $Z$ is exactly the uncertainty in $N$. The chain rule for information entropy giveswhere $h_2$ is binary entropy. HenceAt $\epsilon =1/4$, the result is approximately $0.188722$ bits, although all three pairwise mutual informations are zero. At $\epsilon =1/2$, all eight triples have probability $1/8$, giving joint independence and zero total correlation. At $\epsilon =1$, the complementary integer parity rule is deterministic again, so total correlation returns to one bit.

The example distinguishes "there is a dependence" from "a pair reveals it." Once $X,Y$ are known together, $Z$ is predictable at zero noise; knowing either one alone leaves a fair bit. Total correlation detects the joint departure from independence, but does not by itself specify which variables cause which others.

A probability distribution is non-degenerate when it is not a point mass. This condition excludes constant limits in limit theorems and in the maximum domain of attraction of a distribution function.

Nonparametric statistics studies statistical inference without restricting the unknown probability distribution or target function to a fixed finite-dimensional parametric statistical model. In density estimation and nonparametric regression, the number of features that an estimator can represent may grow with the sample size. Rank tests offer another approach: the probability distribution of their test statistic under the null hypothesis can be obtained without specifying the common continuous probability distribution.

"Nonparametric" does not mean free of assumptions or tuning choices. A kernel density estimator still requires a kernel for density estimation and a smoothing bandwidth, and its interpretation depends on assumptions about the statistical sample. The freedom is useful when a prescribed shape, such as a single normal distribution, would hide features that the observations might reveal.

The normal distribution is a probability distribution on the real line with mean $\mu$ and positive standard deviation $\sigma$. Its probability density function isIn the notation $X\sim N(\mu ,{\sigma }^2)$, the second parameter is the variance ${\sigma }^2$, not the standard deviation $\sigma$. The probability density function is symmetric about $\mu$, highest there, and integrates to one. These formulas and the parameter convention are given in MIT 18.05, Reading 5c, §5: normal distributions.

The normal distribution is a location-scale family: if $Z$ is a standard normal random variable, then $X=\mu +\sigma Z$. Changing $\mu$ moves the center; changing $\sigma$ stretches the horizontal distances from it. The reciprocal factor $1/\sigma$ lowers the probability density function as the curve widens, preserving total area. This is why one standard normal distribution suffices to calculate probabilities for the whole family. ^[ref].

The height $f_X(x)$ is a probability density function value, not $\mathbb{P}(X=x)$. For $a\le b$, the probability $\mathbb{P}(a\le X\le b)={\int }_a^b{f}_X(x)dx$ is an area, and $\mathbb{P}(X=x)=0$ for every individual $x$. If $X$ is measured in some unit, $\mu$ and $\sigma$ use that unit, ${\sigma }^2$ uses its square, and $f_X$ uses its reciprocal; the integral is a dimensionless quantity. A change of measurement unit therefore changes the height without changing the probability. The area interpretation is developed in MIT 18.05, Reading 5b: continuous random variables.

The usefulness of the normal distribution goes beyond its simple shape. Under the central limit theorem, sums of independent and identically distributed random variables with finite positive variance, after centering and scaling, approach the standard normal distribution. Thus approximations by a normal distribution can arise even when the individual random variables have quite different probability distributions. The hypotheses matter: a model of measurement error still needs evidence that this approximation is appropriate. ^[ref]. Exact closure under a linear combination of independent normal random variables provides another reason the family is convenient.

Historically, the curve appeared in a different problem from fitting measurements. In the 1756 edition of The Doctrine of Chances, Abraham de Moivre reproduces work he dates to 12 November 1733 on approximating sums of binomial distribution probabilities for equally likely success and failure; page 245 already gives the one-sided central area as $0.341344$. ^[ref]. In his 1809 Theoria motus, Carl Friedrich Gauss derived a normal distribution for measurement error from a condition on the arithmetic mean, then connected maximizing its likelihood to minimizing the sum of squared errors under that model, the principle behind ordinary least squares. These are distinct contributions: approximating binomial distribution probabilities and justifying an estimation rule. ^[ref].

For a short normalization check, substitute $z=(x-\mu )/\sigma$ in the integral of $f_X$. The Gaussian integral gives ${\int }_{\mathbb{R}}{e}^{-z^2/2}dz=\sqrt{2\pi }$, so the total area is one. The standard normal density is an even function, giving $\mathbb{E}[Z]=0$; integration by parts gives $\mathbb{E}[Z^2]=1$. Then linearity of expectation and the variance scaling rule applied to $X=\mu +\sigma Z$ yield mean $\mu$ and variance ${\sigma }^2$.

A probability distribution assigns total probability one to its sample space. For a nonnegative probability density function this means that its integral over the full domain equals one; for a probability mass function the sum of its masses equals one. A nonnegative integrable function with positive finite integral can be divided by that integral to normalize it. A negative function cannot be a probability density function even if a formal manipulation purports to normalize it.

Suppose $X_1,\dots ,X_n$ are independent and identically distributed with normal distribution $N(\mu ,{\sigma }^2)$, where the mean $\mu$ is unknown and the standard deviation $\sigma >0$ is known. The sample mean $\bar{X}=n^{-1}{\sum }_i{X}_i$ has normal distribution $N(\mu ,{\sigma }^2/n)$ and standard error $\sigma /\sqrt{n}$. With $z_q$ the standard normal quantile at $q$, the displayed confidence interval has exact coverage probability $1-\alpha$ for every $\mu$ and every positive integer $n$. No central limit theorem approximation is needed.

Take $\sigma =2$, $n=16$ and nominal confidence 95%. The standard error is $2/\sqrt{16}=0.5$, and $z_{0.975}=1.9599639845\dots$. ThereforeRounded for reporting, this is $\bar{x}\pm 0.98$. If $\bar{x}=1.2$, the realized confidence interval is approximately $[0.22,2.18]$. Exact 95% calibration refers to the unrounded standard normal quantile; using $1.96$ literally gives coverage probability $0.9500042097\dots$, almost but not exactly 0.95.

The confidence interval's width scales as $n^{-1/2}$: increasing sample size from 16 to 64 halves its half-width to $0.4899909961\dots$. If $\sigma$ is unknown, merely substituting the square root of a sample variance loses the displayed exact calibration; for the same normal distribution model, use a Student t confidence interval.

The figure holds the true mean at $\mu =0$ and independently generates 40 samples of 16 observations from the same normal distribution. NumPy's PCG64 generator with seed 42 gives 39 covering confidence intervals and one miss, at sample 32: observed coverage is $39/40=97.5\%$. Every bar uses the unrounded standard normal quantile. This finite collection illustrates the moving endpoints; it does not establish the analytic coverage probability or require exactly two misses.

The beauty of the construction is that a familiar standard normal distribution calibrates every possible unknown mean at once. For a short proof, the pivotal quantityhas a probability distribution independent of $\mu$. The event $|Z|\le z_{1-\alpha /2}$ has probability $1-\alpha$; rearranging it gives $\mu \in C(X)$. The right panel shows the equivalent event $|\bar{X}|\le 0.9799819923\dots$ when $\mu =0$.

For independent $Z\sim N(0,1)$ and $\Lambda \sim {\chi }_{\nu }^2$, the displayed ratio has the Student t-distribution with $\nu >0$ statistical degrees of freedom. Equivalently, conditional on $\Lambda =\lambda >0$, $T$ has normal distribution $N(0,\nu /\lambda )$. This is a normal scale mixture representation of Student t: a small latent denominator occasionally produces a much larger spread. The result is a heavy-tailed distribution even though every conditional probability distribution is a normal distribution.

For a short derivation, independence lets us multiply the conditional normal density and the chi-squared distribution's probability density function, then integrate over the unobserved $\lambda$:The resulting gamma function integral has shape $(\nu +1)/2$ and rate $(1+t^2/\nu )/2$, yielding the normalized probability density function above. Also, for $\nu >2$, conditional expectation gives $\mathbb{E}[T^2]=\nu \mathbb{E}[{\Lambda }^{-1}]=\nu /(\nu -2)$. Thus unit scale does not mean unit variance. Finally, $\mathbb{E}[\Lambda /\nu ]=1$ and $\mathrm{Var}(\Lambda /\nu )=2/\nu$ imply convergence in probability to 1 by Chebyshev's inequality; Slutsky's theorem then gives the standard normal distribution limit.

A null distribution is the probability distribution of a test statistic under a specified null hypothesis. It calibrates a hypothesis test's test size and p-value. For a simple hypothesis it is one distribution; for a composite null hypothesis it may depend on unknown nuisance parameters.