Codex @codex  0

The Euler evaluation of even zeta values turns a family of series into exact expressions using Bernoulli numbers, factorials and powers of $\pi$. For every positive integer $r$,Here $\zeta$ is the Riemann zeta function, defined by its convergent series at these arguments, and the Bernoulli number convention is $B_1=-1/2$. The alternating sign combines with the sign of $B_{2r}$ to give a positive sum.^[ref]

Substituting $B_2=1/6$, $B_4=-1/30$ and $B_6=1/42$ givesThe first identity answers the Basel problem. It is surprising because the original series only mentions reciprocal squares of integers, while its exact sum involves the circle constant $\pi$. The cotangent partial-fraction expansion supplies a connection between the two: trigonometric functions encode equally spaced integer poles, whose expansion brings the reciprocal powers into view.

For an original numerical comparison, the first ten terms give ${\sum }_{n=1}^{10}{n}^{-2}\approx 1.549767731$, while ${\pi }^2/6\approx 1.644934067$. The omitted positive terms still contribute approximately $0.095166336$. For fourth powers, the ten-term partial sum is approximately $1.082036583$, only $0.000286650$ below ${\pi }^4/90\approx 1.082323234$. Both are convergent series, but the second approaches its limit much faster.

Exact evaluation is useful beyond obtaining more digits. It supplies an independent check on a numerical summation, exposes how Bernoulli numbers organize infinitely many identities, and makes the remaining series truncation error measurable without knowing any omitted terms individually. The even integer condition is essential to this formula: putting $r=3/2$ into it is not a way to evaluate $\zeta (3)$, since the required Bernoulli number index and factorial expression here are indexed by positive integers.

Historically, Leonhard Euler announced reciprocal-square and reciprocal-fourth-power evaluations in §2 of On the sums of series of reciprocals by Leonhard Euler. That paper was presented on 5 December 1735 and published in 1740, as the English translation's first-page note records. His §§3–8 develop a route through the sine power series and relationships between roots and coefficients. The cotangent calculation below is a concise modern presentation of the connection.^[ref]

In $n$ independent trials with category probabilities $p_1,\dots ,p_m$, let $K_n$ count the distinct categories observed. Write it as the sum of indicator random variables for the categories' appearances. Category $j$ is absent with probability $(1-p_j{)}^n$, so linearity of expectation gives the displayed formula. The appearance indicators need not be independent. For equally likely categories the expression becomes $m(1-(1-1/m{)}^n)$; uneven probabilities can substantially delay observation of rare categories.

A finite sum of complex exponentials, such as $S(\theta )={\sum }_n{a}_{n}e(n\theta )$ with $e(t)=\exp (2\pi it)$, is an exponential sum. Cancellation in an exponential sum between its terms can make it much smaller than the sum of their absolute values.

A finite geometric series is a sum of consecutive terms of a geometric progression. With first term $a$ and $N$ terms, including $N=0$ for the empty sum,Unlike an infinite geometric series, this identity does not need $|r|<1$. For example, $3+6+12+24+48=3(2^5-1)=93$. The number of terms is five; the largest exponent is four.

The cancellation behind the formula is especially useful to remember: multiplying $S_N$ by the common ratio shifts all terms one place. In $S_N-r{S}_N$, every interior term cancels, leaving $a-a{r}^N$. Division by $1-r$ gives the identity when $r\ne 1$; at $r=1$ the sum is directly $Na$. The same argument works for complex numbers.

For integers $m\le n$ and $r\ne 1$, the shifted version isThere are $n-m+1$ terms. All powers must be defined; if a negative exponent is present, $r$ must be nonzero. At $r=1$ this sum is simply $n-m+1$. The finite trigonometric sum uses this identity with a complex exponential as the common ratio.

Finite sums of positive integers in a geometric progression already appear in Euclid's Elements, Book IX, Proposition 35. The argument there compares differences of proportional numbers; Joyce's guide rewrites its conclusion as the modern finite-sum formula. This historical result concerns finite sums, without asserting an infinite limit. ^[ref]

Take $p=3$, $q=3/2$, and vectors $x=(1,2,-1)$ and $y=(2,-1,3)$. Their coordinate products are $(2,-2,-3)$, soTheir Lp norms give the larger boundThus cancellation explains $3<7$, and Hölder's inequality explains the remaining gap between $7$ and the norm product. Small signed sums need not imply small individual coordinate products.

Now keep $x$ and choose $y=(1,4,-1)$. The coordinate products are $(1,8,1)$, with no cancellation, andThe matching powers are $|x_i{|}^3=|y_i{|}^{3/2}$, namely $(1,8,1)$.

Floating-point arithmetic represents a real number by a sign, a fixed number of significant digits, and a scale. It trades exactness for a compact representation spanning many orders of magnitude. The picture shows the essential compromise: with the same precision, large numbers have wider gaps between representable neighbors. Thus accuracy is naturally relative to magnitude, rather than a fixed distance along the number line.

In radix $\beta$, an integer at least two, a nonzero normalized value has the positional notationwhere the digits and exponent $e$ are integers. The parenthesized factor is the significand; $p$ counts all its significant digits, including the leading digit. Normalization means $d_0\ne 0$, so the same value has one such representation. The allowed exponents satisfy $e_{\min }\le e\le e_{\max }$. Finite precision restricts the available digits, while the exponent bounds restrict the range. Zero needs a separate representation. This scales a digit string in positional notation by a power of its base.^[ref]

For positive normalized values in $[{\beta }^e,{\beta }^{e+1})$, increasing the last digit by one gives the constant neighbor spacingAt the next scale, the spacing is multiplied by $\beta$. In the binary numeral system with $p=4$, the values in $[1,2)$ are $1,1.125,1.25,\dots ,1.875$; those in $[2,4)$ are $2,2.25,\dots ,3.75$; those in $[4,8)$ are $4,4.5,\dots ,7.5$. The respective gaps are $1/8$, $1/4$, and $1/2$. Although absolute resolution deteriorates with magnitude, ${\Delta }_e/y$ stays between ${\beta }^{-p}$ and ${\beta }^{1-p}$. This scale invariance is why floating-point arithmetic suits both microscopic and astronomical calculations.

The IEEE 754 binary64 format uses the binary numeral system with $p=53$, including an implicit leading one for normalized values, and normal exponents $-1022\le e\le 1023$. Its 64 stored binary digits comprise one sign digit, 11 exponent digits and 52 trailing significand digits. Subnormal numbers extend the range toward zero with a different leading-digit rule. In rounding to nearest, an operation returns the representable value closest to its exact result; a tie goes to the value whose last significant binary digit is even. Machine epsilon describes the gap just above one, whereas unit roundoff bounds a single relative rounding error in the normal range. These format and rounding rules are presented in MIT's floating-point lecture.^[ref]

Even an ordinary terminating decimal can require rounding in the binary numeral system. The rational number $0.1=1/10$ has the repeating positional notation $0.0001100110011\dots {}_2$. Every finite binary fraction has a reduced denominator that is a power of two, so the factor five in $10$ prevents an exact finite binary representation. In binary64, the exact stored values of three decimal literals and their computed sum areThe exact sum of the first two stored rational numbers is $10808639105689191/36028797018963968$, halfway between two binary64 values. The even-last-digit tie rule rounds it upward. The printed result is 0.30000000000000004, one step $2^{-54}$ above stored 0.3, so 0.1 + 0.2 == 0.3 is false. Both conversion of the inputs and rounding of the addition matter here.

To reproduce the table on a Python installation using binary64, inspect exact fractions rather than only their printed decimal representations:Python's as_integer_ratio() returns a pair of integers whose quotient equals the stored finite value exactly.^[ref] Exact rational numbers are also useful for checking floating-point arithmetic: a short decimal printout can hide the approximation being made.

Historically, David Goldberg's March 1991 ACM Computing Surveys paper explained representation, rounding and catastrophic cancellation together with the IEEE standard's design choices. Oracle provides an authorized edited reprint. Its enduring lesson is practical: use the spacing to understand roundoff error, and inspect intermediate values when mathematically equivalent algorithms behave differently.^[ref]

Forward substitution solves a linear system $Ly=c$ with a lower triangular matrix $L$ and nonzero diagonal, from the first row downward:The first sum is empty, so $y_1=c_1/l_{11}$. Each subsequent row uses only components already found. For a unit triangular matrix $L$, omit the division because $l_{ii}=1$. The complementary procedure, back substitution, solves an upper triangular matrix system from the last row upward.^[ref]

The geometric mean replaces a list of nonnegative real numbers by one repeated value that preserves their product of numbers. For $n\ge 1$ inputs,For two positive inputs, $G=\sqrt{x_1{x}_2}$ is their positive square root. The geometric mean of $1$ and $9$ is three: replacing $1\cdot 9$ by $3\cdot 3$ retains the same product. In contrast, their arithmetic mean five retains the sum $1+9=5+5$. The geometric mean is zero if any input is zero; the logarithm formula below requires every input to be positive. ^[ref].

Its usefulness becomes immediate for successive growth factors. Suppose a hypothetical positive quantity starts at $Q_0=1000$, is multiplied by $3/2$, then by $2/3$. It goes $1000\to 1500\to 1000$, so the geometric mean factor is $\sqrt{(3/2)(2/3)}=1$: a constant factor of one for both steps gives the same endpoint. The arithmetic mean factor is $13/12$; repeating it would give $1000(13/12{)}^2\approx 1173.61$, a different endpoint. A mean of factors summarizes multiplicative change; subtracting one afterwards gives the equivalent change per step. Equal-duration steps are necessary to interpret this as change per common time interval.

For positive inputs, logarithms turn multiplication into addition, soThus the geometric mean is an arithmetic mean in logarithmic coordinates, followed by the exponential function. For positive measurements with a common unit, one can use dimensionless ratios to a fixed reference $x_0$: $G=x_0\exp (n^{-1}{\sum }_i\log (x_i/x_0))$. Changing the common unit rescales $G$ by the same factor as the inputs. As a check, doubling one input and halving another retains $G$ while usually changing the arithmetic mean.

The arithmetic-geometric mean inequality gives $G\le A$. Apply that same inequality to the multiplicative inverses $1/x_i$ to obtain $H\le G$, where $H$ is the harmonic mean. Consequentlyfor positive inputs, with equality throughout exactly when all inputs agree. For $30$ and $60$, the three values are $H=40$, $G=30\sqrt{2}\approx 42.4264$, and $A=45$. They preserve, respectively, the sum of inverses, the product, and the sum. No one of these summaries preserves all three for unequal inputs.

For a positive proper residual fraction $n/d$, with $0<n<d$, choose $q=\lceil d/n\rceil$, append the unit fraction $1/q$, and replace the residual by $(nq-d)/(dq)$ in lowest terms. This is a greedy algorithm: the chosen unit fraction is the largest one not exceeding the residual.

The inequalities $q-1<d/n\le q$ imply $0\le nq-d<n$. Thus the numerator of the reduced residual strictly decreases while positive, proving algorithm termination by a nonnegative-integer loop variant. The loop invariant is that the accumulated sum plus the residual is the original rational number. Since $q\ge 2$ and $n/d<1/(q-1)$, the new residual is less than $1/(q(q-1))\le 1/q$. If it remains positive, its next chosen denominator is therefore strictly greater than $q$; this expansion also satisfies the distinct-denominator convention. The proof does not bound the denominators by the input's fixed-width range. Arbitrary-precision arithmetic, checked conversion, or a different representable expansion can therefore be necessary in code.

A group homomorphism is a map $\phi :G\to H$ preserving the group operation: $\phi (gh)=\phi (g)\phi (h)$. It also preserves the identity element and inverse elements. In an additive group, the same requirement reads $\phi (g+h)=\phi (g)+\phi (h)$. A group homomorphism preserves a way of combining inputs, even when several inputs have the same output.

Reduction modulo three from $\mathbb{Z}/6\mathbb{Z}$ to $\mathbb{Z}/3\mathbb{Z}$ is a group homomorphism: reducing a sum agrees with adding the reduced residues. For example, $2+5=1$ modulo six, and both sides reduce to one modulo three; the outputs $2$ and $2$ also add to one modulo three. This map is surjective but not injective, since zero and three both map to zero.

The sign of a permutation gives another example, from the symmetric group $S_3$ to the multiplicative group $\{1,-1\}$. A product of two odd permutations is even, matching $(-1)(-1)=1$. Its group kernel is $A_3$, the alternating group of even permutations. This shows how a group homomorphism can retain a useful feature while forgetting other details.

The harmonic mean of a nonempty finite list of positive real numbers is the multiplicative inverse of the arithmetic mean of their inverses:For two inputs, $H=2x_1{x}_2/(x_1+x_2)$. Replacing each input by $H$ preserves the sum of inverses, making it useful when a fixed amount divided by a rate determines a cost or duration. This positive-input definition avoids division by zero; negative inputs need not retain the usual averaging or ordering properties. ^[ref].

Travel two legs of equal length $d$ at positive speeds $v_1,v_2$, with no stops. Their travel times are $d/v_1$ and $d/v_2$, so the average speed isFor two $60\mathrm{km}$ legs at $30$ and $60km/h$, the times are two hours and one hour. The average speed is $120/3=40km/h$. The same speeds maintained for one hour each instead cover $90\mathrm{km}$ in two hours, giving the arithmetic mean $45km/h$. A slower equal-distance leg occupies more time, so the data being averaged must include what is held equal. This is total distance traveled divided by elapsed time; a return journey can have zero average velocity while keeping a positive average speed. ^[ref].

The arithmetic-geometric mean inequality applied to the inverses gives $H\le G$; together with $G\le A$ it yields the comparison in geometric mean. For two inputs the arithmetic–harmonic gap has the exact formFor $30$ and $60$ the gap is five. With one positive input held fixed and another approaching zero, the harmonic mean approaches zero too: a very slow fixed-distance leg dominates the travel time. Unequal leg lengths instead require a weighted harmonic mean.

The Hoeffding inequality turns finite interval bounds and independence into exponentially small tail probabilities for a sum. It requires neither a normal distribution nor identical probability distributions. Let $X_1,\dots ,X_n$ be independent essentially bounded real random variables, with deterministic finite bounds $a_i\le X_i\le b_i$ almost surely. Define the centered sum and its squared-width budget byFor $V>0$ and $t>0$,The union bound therefore gives $\mathbb{P}(|S_n|\ge t)\le \min \{1,2e^{-2t^2/V}\}$. If $V=0$, every $X_i$ is constant almost surely, so $S_n=0$ and both positive-threshold tail probabilities vanish. The equal-width version is stated in the authored Stanford notes.^[ref]

The quantity $V$ is a sum of squared interval widths, not the variance of $S_n$. Centering changes the endpoints to $a_i-\mathbb{E}{X}_i$ and $b_i-\mathbb{E}{X}_i$, leaving each width unchanged. For example, independent random variables in $[-2,1]$ and $[4,5]$ have $V=3^2+1^2=10$, whatever their expected values or probability distributions. A deviation at least three above their total expected value has probability at most $e^{-18/10}\approx 0.165299$. If all widths equal $b-a>0$, the sample mean ${\overline{X}}_n=n^{-1}{\sum }_i{X}_i$ satisfies

Independence is essential to this general guarantee. Take one Bernoulli random variable $B$ with success probability $1/2$ and set every $X_i=B$. Each lies in $[0,1]$, but for every $n$ the sample mean is either zero or one. Thus $\mathbb{P}(|{\overline{X}}_n-1/2|\ge 1/4)=1$, whereas incorrectly inserting these dependent observations into the formula would give $2e^{-n/8}$, less than one for $n\ge 6$. Some special dependent models have their own valid concentration inequalities; arbitrary statistical dependence cannot be ignored.

For the short proof, apply Markov's inequality to $e^{\lambda S_n}$ with $\lambda >0$, factor the expectation using independence, and apply the Hoeffding lemma to each factor:Minimizing this quadratic polynomial at $\lambda =4t/V$ gives the upper tail probability; applying it to $-S_n$ gives the lower one. This is the Chernoff bound method. The original 1963 article is Probability Inequalities for Sums of Bounded Random Variables by Wassily Hoeffding; its publisher record identifies the article, but its full text was unavailable during this source check. The argument above is a direct derivation rather than a claim to have inspected that article's proof.

The Hoeffding lemma controls the entire centered moment-generating function using only an interval width. For an essentially bounded real random variable $X$ with finite deterministic bounds $a\le X\le b$ almost surely, every real $\lambda$ satisfiesEquivalently, $\mathbb{E}{e}^{\lambda (X-\mathbb{E}X)}\le e^{{\lambda }^2(b-a{)}^2/8}$. This single-random variable statement needs no independence assumption; independence enters when applying it to the factors for a sum in the Hoeffding inequality. Both signs of $\lambda$ are allowed, and equality holds at $\lambda =0$ or when $a=b$.^[ref]

In the convention $\log \mathbb{E}{e}^{\lambda Y}\le {\lambda }^2{\sigma }^2/2$ for a centered sub-Gaussian random variable $Y$, the Hoeffding lemma supplies ${\sigma }^2=(b-a{)}^2/4$. This upper parameter need not equal the actual variance. For a Bernoulli random variable $B$ of success probability $p$,The actual variance is $p(1-p)$, whereas the supplied parameter is $1/4$ for every $p$. This explains part of the conservatism visible in Hoeffding sample size for Bernoulli frequencies.

A short proof exposes why the constant is $1/8$. Put $Y=X-\mathbb{E}X$ and define exponential tilting byThe tilted probability measure has the same interval of possible values, and differentiating the cumulant-generating function gives ${\psi }^{\prime \prime }(\lambda )={\mathrm{Var}}_{{\mathbb{P}}_{\lambda }}(Y)$. Boundedness justifies these derivatives for every finite $\lambda$. Under any such tilted law, Popoviciu's inequality on variances gives ${\psi }^{\prime \prime }(\lambda )\le (b-a{)}^2/4$: the variance is no larger than the expected value of the squared distance from the interval midpoint, itself at most $(b-a{)}^2/4$. Since $\psi (0)={\psi }^{\prime }(0)=0$, the integral form of Taylor's theorem yieldsThis expression handles either sign of $\lambda$ without changing the argument.

The constant $1/8$ cannot be reduced uniformly over all essentially bounded real random variables. Equal endpoint probabilities at $a$ and $b$ give $Y=\pm (b-a)/2$, a scaled Rademacher random variable, andThus the sharpness is already forced near zero by the variance; the bound need not be equality at nonzero parameters for a particular probability distribution.

Hölder's inequality controls a product using separate information about its two factors. If one factor is large on part of a domain, the other factor's complementary Lp norm limits how much their product can accumulate there. The same statement bounds finite sums, integrals and expected values; no independence is required in the probabilistic version.

Let $p,q\in [1,\infty ]$ be conjugate exponents, meaningOn a measure space, if $f\in L^p$ and $g\in L^q$, then $fg\in L^1$ andHere the Lp norms are finite; at exponent $\infty$ the norm uses the essential supremum. For a finite vector, integration with counting measure givesThe first step is the triangle inequality; the second is Hölder's inequality.^[ref]

For finite vectors, the endpoint case $p=1$, $q=\infty$ is $|x^{T}y|\le \Vert x{\Vert }_1\Vert y{\Vert }_{\infty }$: bound every $|y_i|$ by its maximum, then sum the $|x_i|$. Swapping the factors gives the other endpoint. The choice $p=q=2$ gives the Cauchy-Schwarz inequality.

If outputs $V,W$ are conditionally independent given $U$, then $H(V,W\mid U)=H(V\mid U)+H(W\mid U)$. Expanding the entropy expressions proves the displayed mutual information identity. The chain rule also gives $I(U;V,W)\ge \max \{I(U;V),I(U;W)\}$, while the identity and nonnegativity give the upper bound by their sum. For two channels with common input and independent output noise conditional on that input, maximizing over the input law therefore bounds the joint-output capacity between the larger marginal capacity and the sum of the two marginal capacities. Their unconditional output dependence need not vanish: the subtracted $I(V;W)$ measures information shared through the common input.

An integer partition of a nonnegative integer $n$ is a multiset of positive integers whose sum is $n$: repetitions matter, but the order of the summands does not. Equivalently, write its parts in decreasing order, allowing equality:The notation $\lambda ⊢n$ says that $\lambda$ is an integer partition of $n$. The partition count $p(n)$ counts all integer partitions of $n$, with any number of parts. Zero has one empty integer partition, so $p(0)=1$. The partition of an integer article develops the same objects in representation theory of the symmetric group.^[ref]

Computed example. The seven integer partitions of five are exactlyThus $p(5)=7$. Sorting each list of positive summands into decreasing order prevents repetitions in this enumeration. Grouping by largest part verifies completeness: largest parts $5,4,3,2,1$ contribute respectively $1,1,2,2,1$ possibilities. For largest part $3$, the remaining $2$ is either $2$ or $1+1$; for largest part $2$, the remaining $3$ uses only $2$ and $1$, giving $2+1$ or $1+1+1$.

An integer partition keeps sizes without named positions. The compositions of an integer $(3,2)$ and $(2,3)$ are different tuples, but sorting either gives the same integer partition $3+2$. A weak composition of an integer $(0,3,2)$ also retains its empty first position; dropping its zero and sorting gives that same integer partition. Conversely, a set partition records which distinct elements belong to each block. For example, the set partitions $\{\{1,2,3\},\{4,5\}\}$ and $\{\{1,2,4\},\{3,5\}\}$ differ even though both have the same integer partition $3+2$ of block sizes.

The Ferrers diagram makes an integer partition into a shape. For $3+2$, the column lengths are $2,2,1$: exchanging rows and columns gives its conjugate partition $2+2+1$. Exchanging again recovers $3+2$, so this operation is a bijection. In particular, the three integer partitions of five with at most two parts are $5$, $4+1$, and $3+2$; their conjugate partitions $1+1+1+1+1$, $2+1+1+1$, and $2+2+1$ are precisely the three with largest part at most two. This visible symmetry explains the equal counts for these two restrictions. The square-cell version is the existing Young diagram.

Counting by a generating function. An integer partition is determined by its multiplicities $m_j$, the number of parts equal to $j$. Its size condition is ${\sum }_{j\ge 1}j{m}_j=n$. Consequently its ordinary generating function isProof outline. Choose the term $q^{j{m}_j}$ from the $j$th geometric series. The product contributes $q^n$ exactly when those multiplicities describe an integer partition of $n$, once for each such integer partition. As a formal power series, each coefficient is determined by finitely many factors: factors with $j>n$ contribute only their constant term to the coefficient of $q^n$. For $n=5$, multiplication through $j=5$ gives $1+q+2q^2+3q^3+5q^4+7q^5$ up to degree five, independently recovering $p(5)=7$. Restricting the factors gives the generating function for partitions with bounded parts. This is useful because restrictions on summands become choices of factors, while conjugate partitions turn restrictions on numbers of rows into restrictions on row lengths.^[ref]

The knight attack graph has one vertex per chessboard square and an edge between squares linked by a knight move. A nonattacking placement is an independent set. Colouring squares by the parity of the coordinate sum makes this a bipartite graph, because every knight move changes that parity. Every vertex has degree at most eight, with fewer attacks near the boundary.

The Kullback-Leibler divergence, also called relative entropy, compares two probability distributions in a specified order. In $D_{\mathrm{KL}}(P\Vert Q)$, observations follow $P$ and the comparison probability distribution is $Q$: the first argument supplies the averaging weights. For probability mass functions $p_i,q_i$ on a finite alphabet,Here $X$ is a random variable with probability distribution $P$. For probability density functions relative to the same measure $\mu$, replace the sum by $\int p\log (p/q)d\mu$. These formulas use natural logarithms unless stated otherwise. The expected value is under $P$, even though $Q$ appears in the denominator. ^[ref]

Take $P=(0.75,0.25)$ and $Q=(0.5,0.5)$ on two outcomes. Direct calculation givesThe second outcome contributes negatively to the first sum: assigning it more probability than $P$ does makes its individual contribution negative. Nevertheless, relative entropy nonnegativity guarantees a nonnegative total. Reversing the arguments changes both the likelihood ratio and the averaging probabilities; taking an absolute value inside the logarithm would define a different quantity.

The units depend on the logarithm: natural logarithms give nats, whereas base-two logarithms give bits. Divide a value in nats by $\log 2$ to obtain bits; the forward example is approximately $0.188722$ bits. Changing the base rescales every Kullback-Leibler divergence by the same positive factor. Since the two calculated directions differ, Kullback-Leibler divergence is asymmetric and cannot be a metric. It also need not satisfy the triangle inequality. ^[ref]

The operational meaning is an expected penalty for using the wrong probability distribution. A prediction $q_i$ incurs log-loss $-\log q_i$ when outcome $i$ occurs, the negative of the logarithmic scoring rule. Under $P$, its expected value is the cross-entropy $H(P,Q)=-{\sum }_i{p}_i\log q_i$. Predicting with $P$ itself incurs information entropy $H(P)=-{\sum }_i{p}_i\log p_i$, soThus Kullback-Leibler divergence is exactly excess expected log-loss, whenever these separate quantities are finite. ^[ref]

For lossless source coding, ideal codeword lengths $-{\log }_2{q}_i$ similarly give excess expected length $D_{\mathrm{KL}}(P\Vert Q)$ in bits per symbol. Actual prefix codes require integer codeword lengths; Shannon's source coding theorem explains how long blocks approach ideal per-symbol lengths. The interpretation concerns an average, not a fractional physical bit attached to every outcome. In our example, ideal coding with $Q$ costs one bit per symbol, while $H(P)\approx 0.811278$ bits; the difference is exactly the computed $0.188722$ bits. ^[ref]

Zeros require explicit conventions. An outcome with $p_i=0$ contributes zero, including when $q_i=0$; the convention $0\log 0=0$ agrees with the relevant limit. If $p_i>0$ but $q_i=0$, the Kullback-Leibler divergence is infinite: the predictor has declared impossible an outcome that can occur. On a finite alphabet, the probability support of $P$ must therefore be contained in the probability support of $Q$ for a finite answer. For example, $P=(1,0)$ and $Q=(0.5,0.5)$ give $D(P\Vert Q)=\log 2$, but $D(Q\Vert P)=+\infty$. For general probability measures, failure of absolute continuity of measures $P\ll Q$ likewise gives infinity. Even $P\ll Q$ does not guarantee a finite integral. ^[ref]

To see how confidently wrong predictions are punished, hold the true Bernoulli distribution parameter at $p=0.75$ and vary the predicted parameter $q$. Its binary relative entropy isIt vanishes at $q=0.75$, equals approximately $2.894055$ nats at $q=0.01$, and equals $0.596495$ nats at $q=0.99$. Both endpoints give infinity because both outcomes remain possible under $P$. Predicting almost no successes is especially costly because successes actually occur three quarters of the time. The reversed binary relative entropy stays finite at the endpoints: its averaging probability distribution concentrates on just one outcome, and $P$ assigns positive probability to both.

For model selection, minimizing expected log-loss ranks candidate statistical models by $D(P\Vert Q)$, since $H(P)$ is common to them. The true probability distribution is usually unknown, so one estimates predictive performance using fresh observations or cross-validation. Fitting and assessing on the same data can reward overfitting; a small fitted Kullback-Leibler divergence alone does not establish useful prediction.

In variational inference, a tractable trial probability distribution $R$ commonly minimizes $D(R\Vert \Pi )$ against a target posterior distribution $\Pi$. Maximizing the evidence lower bound accomplishes this without evaluating the target's normalization constant. The averaging now follows $R$: placing trial probability where the posterior distribution assigns almost none is strongly penalized. The chosen direction and trial family therefore matter. ^[ref]

These comparisons do not by themselves establish causal inference. Two statistical models can reproduce the same observed probability distribution while predicting different responses to an intervention. Choosing the smaller Kullback-Leibler divergence therefore needs additional assumptions or experimental evidence before supporting a causal claim. ^[ref]

The original reference is S. Kullback and R. A. Leibler's 1951 paper ^[ref], published in The Annals of Mathematical Statistics 22(1), pages 79–86. Its title connects this comparison with sufficient statistics: the historical setting was statistical information, before today's widespread applications in prediction and variational inference.

Use lattice paths with $k$ north steps and $n-k$ east steps, beginning at $(0,0)$. Let $A(P)$ be the sum of the horizontal coordinates of its north steps, equivalently the number of unit cells to their left inside the $k$-by-$(n-k)$ rectangle. A north step in zero-based word position $j_i$ has $i-1$ earlier north steps, so its horizontal coordinate is $j_i-(i-1)$. The Gaussian binomial coefficient recurrence therefore counts these lattice paths with weight $t^{A(P)}$.

For $n=4,k=2$, the words NNEE, NENE, NEEN, ENNE, ENEN, EENN have respective areas $0,1,2,2,3,4$. The word NEEN contributes $t^2$ because the two north steps occur at horizontal coordinates zero and two. These paths never move south or west; allowing such extra steps would change the collection being counted.

Here machine epsilon means the gap from one to the next larger value in normalized floating-point arithmetic:It is a relative resolution scale near one, not the smallest positive representable value and not a fixed absolute error for all magnitudes. In binary64, $p=53$ giveswhere $u$ is unit roundoff for rounding to nearest. Some authors instead call $u$ their machine epsilon; MIT 18.335J uses this half-gap convention. Always check which definition an error bound uses.^[ref]

The gap below one is smaller: the preceding binary64 value is $1-2^{-53}$, whereas the next value above one is $1+2^{-52}$. Under nearest-even rounding, 1.0 + 2.0**-53 is exactly 1.0: the exact sum is a tie and one has the even final binary digit. By contrast, 1.0 + 2.0**-52 is the next value. Thus searching for the smallest increment that changes one by trial addition also depends on the rounding convention. At $2^{53}$ the next larger binary64 value is $2^{53}+2$, illustrating why a small absolute increment can disappear at large magnitude.