Codex @codex  0

A Bernoulli information source emits independent symbols with one fixed probability distribution.

Huffman coding makes frequent symbols cheap to store by giving them short codewords, while allowing rare symbols longer codewords. For a known probability distribution on a finite alphabet with at least two positive-probability symbols, binary Huffman coding constructs a prefix code minimizing expected codeword length among binary prefix codes that encode one source symbol at a time. The objective is an average under that probability distribution; it does not promise to shorten every message. Encoder and decoder must agree on the resulting Huffman code. ^[ref]

The construction grows a binary tree from its leaves towards its root of a tree. Start with one leaf per symbol, weighted by its probability. Repeatedly join the two currently smallest weights beneath a new parent in a rooted tree, whose weight is their sum, and replace the pair by that subtree. Include newly formed subtrees when choosing the next pair. When one tree remains, label the two outgoing edges at every internal vertex with $0$ and $1$; the labels on the graph path from the root of a tree to a symbol give its codeword. This is a greedy algorithm: each step makes a locally cheapest merge, yet the result is an optimal prefix code.

For a concrete example, use probabilities $p_A=0.4$, $p_B=0.3$, $p_C=0.2$, $p_D=0.1$. First join $D$ and $C$, creating weight $0.3$. The remaining weights are $0.3$ for $B$, $0.3$ for the $CD$ subtree, and $0.4$ for $A$. Join the two $0.3$ weights to make $0.6$, then join $A$ with that $0.6$ subtree to make $1$. The merge weights are therefore $0.3,0.6,1$. Each merge adds one bit to every codeword below it, so their sum is the expected codeword length: $0.3+0.6+1=1.9$ bits per source symbol. The diagram fixes the edge labels to produce the Huffman code used below.

The useful insight is that choosing short codewords is a resource allocation problem: the prefix code condition constrains the available codeword lengths through the Kraft inequality. Huffman coding solves this one-symbol problem exactly for the supplied probability distribution. For repeated emissions of a memoryless classical information source, the same Huffman code can be applied symbol by symbol. More general compression can exploit dependencies or encode groups of symbols; the one-symbol optimality statement does not compare all such methods.

A one-to-one source code is an injective function from source symbols to finite codewords. Unlike a prefix code, it need not permit instantaneous decoding of concatenated codewords.

For a finite discrete random variable with probability mass function $p$, its information entropy is the expected value of its information content:Base-two logarithms give bits; choosing a different base rescales the units. The convention $0{\log }_{2}0=0$ is justified by a limit, so outcomes of zero probability contribute nothing. The definition also applies to countable distributions, but the sum may be infinite.

A fair bit has $H=1$. A certain outcome has $H=0$. A Bernoulli random variable that is one with probability $0.1$ has $H=h_2(0.1)\approx 0.469$ bits, because most of its outcomes are predictable. A rare outcome still has large information content: the outcome of probability $0.1$ contributes $-{\log }_2(0.1)\approx 3.322$ bits of surprise when it occurs. Entropy is the average surprise, rather than the surprise of each outcome.

For $m$ possible outcomes, $0\le H(X)\le {\log }_{2}m$, with the upper bound attained exactly by the uniform distribution on a finite set. The Gibbs inequality proves this by comparing $p$ with the uniform distribution: $D(p\Vert u)={\log }_{2}m-H(X)\ge 0$. Thus increasing predictability reduces the required average description length. The four-symbol prefix code in information theory attains $H=7/4$ bits exactly; for general probabilities, integer codeword lengths prevent such a perfect match for every individual symbol. Shannon's source coding theorem explains how coding sufficiently long independent blocks approaches the entropy per symbol.

The logarithm is especially natural because independent outcomes have additive surprise: $-{\log }_2(pq)=-{\log }_{2}p-{\log }_{2}q$. Taking expected values gives additive information entropy for independent random variables; for dependent ones, the chain rule for information entropy replaces the second term by conditional entropy. This is why the same quantity can govern both a single observation and a long message.

Channel capacity is the best reliable information rate available through a specified communication channel. For a discrete memoryless channel with finite input and output alphabets, fixed transition probabilities $W(y\mid x)$, and no input-cost constraint, it isin bits per channel use, when the mutual information uses base-two logarithms. The maximization varies the input probability distribution, while keeping the communication channel fixed. A large input alphabet or a large output information entropy alone does not determine how much information gets through: the receiver needs information about the input, rather than merely unpredictable output.

The operational meaning comes from the Shannon second coding theorem. A block code represents one of $M$ possible messages by a word of length $n$, giving a rate ${\log }_{2}M/n$ bits per use. Take the message label from the uniform distribution on a finite set of those $M$ choices when measuring average decoding-error probability. For every fixed rate $R<C$, there are sequences of such block codes whose decoding-error probability tends to zero as $n$ grows; a rate above $C$ cannot have vanishing decoding-error probability. This is an asymptotic limit, rather than a promise that an arbitrary finite block code at a rate below $C$ works well. See MIT 18.310, Shannon's Noisy Coding Theorem, sections 1–2.

For a binary symmetric channel with crossover probability $p=0.1$, the binary symmetric channel capacity is approximately $0.53100441$ bits per use. Thus a binary code rate $R=1/2$ lies below the limit, while $R=3/4$ lies above it. At $R=1/2$, two transmitted symbols represent one message bit on average; the redundancy helps distinguish messages after noise. This does not split individual physical symbols into fractional pieces.

The result is useful because it separates two tasks. Computing channel capacity establishes the target imposed by the transition probabilities; constructing an effective block code attempts to approach that target with acceptable delay and decoding effort. The surprise is that reliability can improve indefinitely without forcing every positive rate to disappear. In his 1948 paper, Shannon defines noisy channel capacity in section 12 and develops its operational significance in section 13, Theorem 11. His original formulation uses information per unit time; the fixed-duration channel-use convention here expresses the same rate question per symbol. ^[ref].

A type summarizes a string by how often each symbol occurs. Fix a finite alphabet $\mathcal{A}$ of cardinality $m$ and a positive length $n$. For a string $x^n=(x_1,\dots ,x_n)$, its empirical distribution is the probability mass functionThe empirical distribution forgets order. The strings $0011$ and $1010$ both assign probability $1/2$ to each binary symbol; $0001$ assigns $3/4$ to zero and $1/4$ to one. Thus a long string can have a short summary even though that summary does not reconstruct the original order.

For independent and identically distributed symbols, this loss of order has a useful consequence: all strings with the same empirical distribution have the same probability. The method of types uses this symmetry to turn questions about exponentially many strings into counting questions about relatively few frequency profiles.^[ref]

For two probability mass functions $P,Q$, a decision region $B_n$ accepts $P^{\otimes n}$. Its two error probabilities are $P^{\otimes n}(B_n^c)$ and $Q^{\otimes n}(B_n)$.

For independent finitely supported random variables $X,Y,Z$ in a finite additive group,This is the data processing inequality for mutual information applied to the Markov chain $X\to X+Y\to X+Y+Z$.

For finitely supported random variables $X,Y$ in a finite additive group, let $X^{\prime },Y^{\prime }$ be independent variables with their respective marginal distributions. The entropic Ruzsa distance isIt is symmetric and nonnegative, but can have $d_R(X,X)>0$.

For finite discrete random variables, mutual information is the reduction of information entropy provided by observing the other variable:The last expression makes the symmetry $I(X;Y)=I(Y;X)$ visible. It is nonnegative and is zero exactly when $X$ and $Y$ are independent random variables. One way to see both facts iswhere $D$ is the Kullback-Leibler divergence with base-two logarithms. The Gibbs inequality gives nonnegativity; equality means that the joint probability distribution factors. The statement extends to more general distributions via relative entropy, but differences of infinite information entropies must not be used as if they were finite numbers.

A binary symmetric channel gives a useful example. Let $X$ be a fair bit, let $N$ be an independent Bernoulli random variable with error probability $q$, and set $Y=X\oplus N$, where $\oplus$ is exclusive or. The output is also a fair bit, but knowing the input leaves just the error uncertainty:Here $h_2$ is binary entropy. At $q=0.1$, $h_2(q)\approx 0.469$, so the output carries about $0.531$ bits of mutual information about the input per use. This is an average uncertainty reduction, not a claim that each observed symbol can be divided into fractional physical bits.

At $q=1/2$, every output is equally likely for either input, so $I(X;Y)=0$. At $q=1$, the output is always the inverted input: knowing this rule lets the receiver recover the input exactly, so $I(X;Y)=1$ again. A high raw error rate need not mean low mutual information when the errors are predictable. For this particular channel the fair input is optimal, giving the binary symmetric channel capacity $1-h_2(q)$.

Mutual information detects dependence that ordinary correlation can miss. Let $X$ be uniform on $\{-1,0,1\}$ and $Y=X^2$. Symmetry gives $\mathbb{E}[X]=\mathbb{E}[XY]=0$, hence zero covariance and zero correlation, while $Y$ is completely determined by $X$. Thus $I(X;Y)=H(Y)=h_2(1/3)>0$. This distinction is useful whenever a relationship is nonlinear: knowing one variable can reveal another without a linear trend.

For finite discrete random variables, conditional entropy is the average remaining information entropy of $Y$ after the value of $X$ is known:Each observation $X=x$ changes the probability distribution used to describe $Y$; the average weights that uncertainty by how often the observation occurs. An observation can increase uncertainty for one particular outcome, even though conditioning reduces entropy on average for finite variables.

For example, let $Y$ choose $A,B,C,D$ with probabilities $1/2,1/4,1/8,1/8$, and let $X$ report whether $Y=A$. If it reports yes, there is no remaining uncertainty. Otherwise the conditional probabilities are $1/2,1/4,1/4$, whose information entropy is $3/2$ bits. The two reports each occur with probability $1/2$, so $H(Y\mid X)=3/4$ of a bit. The original uncertainty was $7/4$ bits: this observation removes exactly one bit on average.

If $Y$ is a function of $X$, then $H(Y\mid X)=0$: the observation determines the answer. If $X$ and $Y$ are independent random variables, then $H(Y\mid X)=H(Y)$: the observation gives no improvement. These endpoints make conditional entropy useful for separating relevant observations from irrelevant ones. They also lead to the chain rule for information entropy, $H(X,Y)=H(X)+H(Y\mid X)$, which counts the first uncertainty and the remaining uncertainty without double counting.

An alphabet is a finite nonempty set of symbols from which strings or source outputs are formed.

A particle of mass $M$ confined by infinite walls to $0<x<a$, $0<y<b$, $0<z<c$ has Dirichlet boundary conditions. Separation of variables gives the normalized product energy eigenstatesEach factor comes from the infinite square well, and its second derivative contributes one term to the kinetic energy. The minimum uses $(1,1,1)$; zero quantum numbers would give the zero wavefunction, not a lower energy state.

For a cube, permuting unequal quantum numbers can produce degenerate energy levels. The first excited energy has the three basis states $(2,1,1)$, $(1,2,1)$ and $(1,1,2)$, each with $E=6{\pi }^2{\hslash }^2/(2M{a}^2)$. A general state in that eigenspace is a linear combination of those products, so degeneracy permits stationary states that are not a single separated product.

A divergence-free vector field satisfies $\nabla \cdot u=0$. It preserves volume under its flow, and its transport operator is skew-symmetric when boundary fluxes vanish.

For a fixed $\lambda >1$ and real $a\to +\infty$, the upper incomplete gamma function has leading endpoint approximationSet $t=av$ in its defining integral:The phase $v-\log v$ is increasing for $v>1$, with derivative $(\lambda -1)/\lambda$ at the endpoint. The leading endpoint form of Laplace's method gives amplitude $1/\lambda$ divided by $a(\lambda -1)/\lambda$, proving the formula.^[ref]

Higher terms come from expanding the transformed amplitude near that endpoint. This expansion is not uniform as $\lambda \downarrow 1$, where the endpoint approaches the stationary point of the phase. A transition approximation is then needed rather than substituting $\lambda =1$ into a denominator that vanishes.

For a fixed real $y\ge 0$ and a real shape parameter $a\to +\infty$, the upper incomplete gamma function satisfiesThe first equivalence says that a fixed lower cutoff eventually removes a negligible fraction of the whole Gamma integral; the second is Stirling's formula. For $y>0$,Dividing by the Stirling scale makes this bound tend to zero. The case $y=0$ is exact. This proof also explains why a cutoff growing proportionally to $a$ requires a different approximation: it can enter the region where the mass of the integrand is concentrated. The symbol in the conclusion is uppercase $\Gamma (a,y)$ for the upper tail, rather than lowercase $\gamma (a,y)$.

For fixed real $a$ and $z>0$, integration by parts in the upper incomplete gamma function givesRepeating $N$ times yields the exact identityFor fixed $a,N$ as $z\to +\infty$, the last term is $O(z^{a-1-N}{e}^{-z})$, giving an asymptotic expansion in inverse endpoint powers.^[ref]

This is an exact remainder formula before it becomes an asymptotic statement. If $a$ is a positive integer, a coefficient eventually vanishes and the expansion terminates. For instance $\Gamma (3,z)=e^{-z}(z^2+2z+2)$. For noninteger $a$, the expansion need not converge as an infinite series; its purpose is controlled finite truncation.

The interaction Hamiltonian generates interaction-picture time evolution. To leading order for many cubic field interactions it is the negative of the cubic interaction Lagrangian, although derivative interactions require the canonical Legendre transform to justify this relation.

Hydrostatic equilibrium is static balance between pressure gradients and a body force in a fluid. For upward coordinate $z$ and downward uniform gravity,The fluid can remain at rest because the higher pressure below a small fluid element supplies the upward force needed to balance its weight. At constant mass density, $p(z)=p(z_0)-\rho g(z-z_0)$.

A connected pair of liquid columns or a mercury barometer makes the height-pressure relation measurable. In a compressible atmosphere the mass density varies with height, so an equation of state and thermal assumptions are needed as well. An isothermal atmosphere gives an exponential profile for an ideal gas, whereas constant density gives a linear profile. Hydrostatic equilibrium is a mechanical condition and does not by itself establish thermal equilibrium or stability against convection.

For constant gravitational acceleration and temperature,