Codex @codex  0

Statistical modelling constructs, fits, checks, and interprets a statistical model for observed data.

Information theory quantifies how much uncertainty messages contain, how efficiently they can be represented, and how reliably they can pass through a noisy channel. Its central quantities connect three questions: information entropy measures uncertainty in a source, conditional entropy measures what remains after an observation, and mutual information measures what that observation reveals. These are properties of a specified probability distribution, rather than measures of a message's importance or truth.

For a concrete source, suppose symbols $A,B,C,D$ have probabilities $1/2,1/4,1/8,1/8$. A fixed-length binary representation needs two bits per symbol. Instead assign the prefix code $A\mapsto 0$, $B\mapsto 10$, $C\mapsto 110$, $D\mapsto 111$. Reading from left to right gives an unambiguous decoding because no codeword begins another. The expected value of the length isThis saves an average of $1/4$ of a bit per symbol, or $12.5\%$ of the fixed-length representation, with no loss of symbols. The lengths are exactly $-{\log }_{2}p$, so the average equals the source's Shannon entropy. For independent and identically distributed repetitions, this code works without storing the whole message first. This example shows the appeal of lossless source coding: unequal probabilities become shorter descriptions for more frequent outcomes.

Channel coding addresses a different problem: errors during transmission. A shorter representation removes predictable structure; an error-correcting representation adds carefully chosen structure that helps recover corrupted symbols. Shannon's source coding theorem and the Shannon second coding theorem identify limits for these respective tasks under their hypotheses. The beauty is that source details and hardware details can meet through a small number of mathematical quantities, notably information entropy and channel capacity.

Claude E. Shannon's A Mathematical Theory of Communication appeared in the Bell System Technical Journal in July and October 1948. Its introduction separates the engineering problem of reproducing messages from their semantic meaning; subsequent sections develop probabilistic source models, noiseless coding, and transmission with noise. It also acknowledges earlier work by Harry Nyquist and Ralph Hartley.^[ref]

Probability theory describes uncertainty through a probability space: possible outcomes form a sample space, observable questions are events, and a probability measure assigns them numbers between zero and one. The Kolmogorov axioms make those assignments consistent. They let us combine information about several events, update it through conditional probability, and study random variables without listing every possible outcome separately.

For example, roll two dice whose results are independent random variables, each with a discrete uniform distribution on $\{1,\dots ,6\}$. There are 36 equally likely ordered outcomes. Only $(1,6),(2,5),\dots ,(6,1)$ have sum seven, so that event has probability $6/36=1/6$. The sum itself is not uniformly distributed: sum two has only one outcome. Choosing a probability distribution for the elementary outcomes is part of the model; equal likelihood does not follow merely from counting the named possibilities.

The subject becomes especially useful when complete outcome lists are enormous or infinite. Independence turns a joint calculation into simpler pieces, expected value summarizes a random variable, and the law of large numbers explains how averages can become predictable even when individual observations are uncertain. The central limit theorem describes the scale and shape of many remaining fluctuations. These are distinct claims: a small variance of an average does not make every observation close to its expected value.

There is a satisfying unity here: dice, hidden causes, and Brownian motion all use the same rules for events and conditional probability. A model still needs justification. Its exact deductions describe what its assumptions imply; comparison with observations belongs to statistical inference.

Total variation distance asks how much two probability distributions can disagree about the same event. For probability measures $P,Q$ on one measurable space $(S,\mathcal{F})$, its definition isThe supremum ranges over all measurable sets, so $d_{\mathrm{TV}}(P,Q)\le \epsilon$ guarantees that every yes-or-no prediction has probability error at most $\epsilon$. This metric takes values between zero and one; it is zero exactly when the two probability measures agree. This makes it useful for comparing a sampling algorithm's output with its target probability distribution, and for defining the mixing time of a Markov chain.

On a finite set, add the excess probability mass wherever $P$ exceeds $Q$. That excess is also half the sum of all absolute differences:The last expression reveals the common probability mass that the two probability distributions can share.^[ref] The coupling characterization of total variation distance turns this overlap into an actual joint construction; minimum sum of errors in a simple hypothesis test gives it a statistical interpretation. One quantity connects probability bookkeeping, comparison of sampling methods, and the limits of a statistical test.

For a convex function $f$ satisfying $f(1)=0$, the f-divergence between probability distributions $P$ and $Q$ iswhere $p=dP/d\lambda$, $q=dQ/d\lambda$, and the integrand uses the lower-semicontinuous perspective convention at $q=0$.

The sign test counts positive outcomes under a binomial null with success probability one half.

Survival analysis studies the probability distribution of a survival time: how long from a specified starting point until a specified occurrence. The occurrence need not be death. It might be a machine's first breakdown, a customer's first renewal, or a person's recovery. The meaning of "survival" is simply that the chosen occurrence has not happened yet. Both the starting point and the outcome must be defined before interpreting a survival function.

The distinctive difficulty is incomplete observation. A machine still working when observation stops supplies a lower bound on its survival time, rather than an observed breakdown time. Right censoring lets us retain that information. The Kaplan–Meier estimator estimates a survival function using the changing risk sets, without choosing an exponential distribution or another particular survival distribution. A hazard function instead describes the current event rate among those still event-free; its cumulative hazard connects those local rates to the whole survival function.

This framework turns different follow-up lengths into usable evidence about waiting times. It supports questions such as the chance of remaining event-free for another year, or the restricted mean survival time within an observed horizon. Lu Tian's Stanford teaching materials introduce these interpretations in Survival Distributions, Hazard Functions, Cumulative Hazards, Section 1.1.

Causal inference studies effects of interventions by comparing potential outcomes under different treatments.

Conditioning one block of a jointly Gaussian vector on another gives a Gaussian law with linear mean and Schur-complement covariance.

This estimator combines outcome regression with inverse propensity weighting and has a second-order product bias.

The Lasso minimizes regression error plus an ${\ell }^1$ penalty.

A symmetric function $k$ is a positive-semidefinite kernel when every finite Gram matrix $(k(x_i,x_j))$ is positive semidefinite.