Codex @codex  0

For a positive random variable $X$, let $D\in \{1,\dots ,9\}$ be its first nonzero base-ten digit. The Benford law is the discrete probability distribution with probability mass functionHere $X$ need not be an integer: $0.0042$, $4.2$ and $4200$ all have $D=4$. Multiplication by an integer power of ten changes the decimal point's position and leaves the first nonzero digit unchanged. Zero has no first nonzero digit, so exclude it. For negative inputs one may explicitly use their absolute values, still excluding zero; that convention must be stated rather than applying a logarithm to a negative number.

The Benford law favors small digits: $\mathbb{P}(D=1)={\log }_{10}2\approx 0.301029996$, whereas $\mathbb{P}(D=9)={\log }_{10}(10/9)\approx 0.04575749$. Thus the model assigns about $30.10\%$ to digit one and $4.58\%$ to digit nine. The masses are positive and their normalization is a finite telescoping series:

Why can digit one be so common? In a decimal significand interval $[1,10)$, the digit-one region $[1,2)$ occupies a larger fraction of the logarithmic scale than the digit-nine region $[9,10)$. The precise assumption is that the fractional part of the base-ten logarithm is distributed uniformly on $[0,1)$; Benford law from logarithmic uniformity turns those unequal interval lengths into probabilities. This makes a surprising digit pattern into a simple geometry of multiplicative size. The logarithmic explanation is also presented in Peter Sheridan Dodds, Benford's Law, essential story.

The Benford law is useful as a proposed model for leading digits when the mechanism supports logarithmic uniformity or an appropriate limiting argument, such as Benford frequencies for powers of an integer. Merely spanning many powers of ten does not establish that assumption. A discrepancy from the Benford law can prompt investigation of the model and data collection; by itself it does not establish fraud, and agreement does not establish authenticity.

Simon Newcomb's 1881 paper, Note on the Frequency of Use of the Different Digits in Natural Numbers, appeared in the American Journal of Mathematics 4, pp. 39–40. Its opening connects unequal leading-digit frequencies with the faster wear of early pages in logarithm tables. The publication metadata and original opening excerpt are available in Newcomb (1881), original publication record and ^[ref]. Frank Benford's The Law of Anomalous Numbers, Proceedings of the American Philosophical Society 78 (1938), pp. 551–572, likewise discusses worn logarithm tables on p. 551 and tabulates observations on p. 553. See Benford (1938), transcription of the original paper with page numbers and publication metadata. These observations motivated study of the digit pattern; they do not make the Benford law universal.

The characteristic function of a real random variable $X$ isIt determines the probability distribution of $X$.

Here $t\in \mathbb{R}$ and $i^2=-1$. By Euler's formula, its real part and imaginary part are $\mathbb{E}[\cos (tX)]$ and $\mathbb{E}[\sin (tX)]$. The complex exponential has absolute value one, so this expected value exists for every real random variable, even without finite absolute moments. Moreover,The bound uses the triangle inequality for a vector-valued integral; the last identity is complex conjugate symmetry.^[ref] Consequently the real part is an even function and the imaginary part is an odd function.

For a discrete probability distribution with values $x_j$ and probabilities $p_j$, the characteristic function is ${\sum }_j{p}_j{e}^{it{x}_j}$: a weighted average of points on the unit circle in the complex plane. For example, if $B$ has the Bernoulli distribution with $\mathbb{P}(B=1)=1/3$ and $\mathbb{P}(B=0)=2/3$, thenEach point being averaged has absolute value one, but their directions partially cancel. At $t=\pi$, the same average is $1/3$; at $t=2\pi$, it returns to one. The characteristic function is thus neither a probability density function nor a probability mass function: its argument is a frequency parameter, and its values can be complex numbers. This Bernoulli distribution has no probability density function with respect to Lebesgue measure, yet its characteristic function is defined at every frequency.

For a probability density function $f$, the same definition becomes ${\phi }_X(t)={\int }_{\mathbb{R}}{e}^{itx}f(x)dx$, the Fourier transform with the displayed sign convention. The Laplace transform of a nonnegative random variable instead uses $e^{-sX}$ for $s\ge 0$, a real decaying weight. For the Bernoulli distribution above, it is $2/3+e^{-s}/3$; neither transform is the probability density function. The uniqueness theorem for characteristic functions explains why the oscillating averages still retain the entire probability law, while the characteristic function of a sum of independent variables turns combining laws into multiplication. These two properties make the representation useful even when directly computing a sum's probability distribution is cumbersome.

A discrete probability distribution is a probability distribution concentrated on a finite or countable set of possible values $S$: a random variable $X$ with this law satisfies $\mathbb{P}(X\in S)=1$. Its probability mass function assigns a probability to each value:Thus the probability of an event is obtained by adding masses, rather than integrating a probability density function. The possible values need not be equally spaced, or even integers. When the relevant moments are finite, its expected value and variance are

For a constructed example, assign values $0,2,5$ masses $0.5,0.3,0.2$. They sum to one; $\mathbb{P}(X\ge 2)=0.3+0.2=0.5$, $\mathbb{E}X=0(0.5)+2(0.3)+5(0.2)=1.6$, and $\mathbb{E}[X^2]=4(0.3)+25(0.2)=6.2$. Hence the variance is $6.2-1.6^2=3.64$. A mean can lie between possible values: no outcome here equals $1.6$.

The useful organizing question is what an experiment records. One yes/no indicator random variable gives a Bernoulli distribution; a success count in a fixed number of independent attempts gives a binomial distribution; the attempt number of the first success gives a geometric distribution. A Poisson distribution describes an event count under different assumptions. Naming the quantity first helps prevent selecting a familiar formula for the wrong experiment.

The one-sample Kolmogorov-Smirnov test checks goodness of fit by comparing an empirical distribution function $F_n$ with a specified cumulative distribution function $F_0$. Its Kolmogorov-Smirnov statistic $D_n$ is the largest vertical gap anywhere between the two curves. Every threshold asks the same concrete question: what proportion of the observations are at or below it, and what probability does the proposed probability distribution assign there? A large discrepancy supplies evidence against the null hypothesis.

For example, four stipulated observations $(1/10,2/5,4/5,9/10)$ compared with the uniform distribution on $[0,1]$ give $D_4=3/10$. Just before $4/5$, only two observations have been counted, so the empirical distribution function approaches $1/2$, while $F_0$ approaches $4/5$. The Kolmogorov-Smirnov statistic section calculates every jump-side gap. These are exact illustrative numbers, not experimental measurements or a claim of statistical significance.

The ordinary one-sample calibration assumes independent and identically distributed real observations and a simple hypothesis $H_0:F=F_0$, with $F_0$ a continuous function fixed before inspecting these observations. All statistical parameters of $F_0$ must be specified.^[ref] Under that null hypothesis, the probability integral transform gives $U_i=F_0(X_i)$ with the uniform distribution on $(0,1)$, retaining independence. Consequently the sampling distribution of $D_n$ depends on the sample size $n$, but not on the particular continuous $F_0$. This is the useful sense in which the calibration is distribution-free.

For the exponential distribution of rate one, $F_0(x)=1-e^{-x}$ for $x\ge 0$. Take $x_i=-\log (1-u_i)$ for the four stipulated $u_i$ above. The probability integral transform changes the horizontal coordinates but preserves both the order statistics and their cumulative heights, so the same Kolmogorov-Smirnov statistic $3/10$ appears on both scales. More generally, for continuous $F_0$, the jump-side formulas depend only on $F_0(X_{(i)})$. Strict increase everywhere is unnecessary: intervals on which $F_0$ is constant have probability zero under the null hypothesis. This explains the invariance without assuming a strictly positive probability density function.

The observed Kolmogorov-Smirnov statistic is a discrepancy, whereas its p-value is a tail probability under the null hypothesis:Here $D_n^{\mathrm{new}}$ is computed from a fresh sample of the same sample size. Choose a significance level $\alpha$ before the test and reject when the calibrated p-value is at most $\alpha$. The value $D_4=3/10$ alone supplies neither a p-value nor a rejection decision. Failure to reject does not establish that the null hypothesis is true. A finite-sample calibration and the asymptotic Kolmogorov-Smirnov theorem are distinct; a cutoff for $D_n$ must be rescaled before use with $\sqrt{n}{D}_n$.^[ref]

Fitting statistical parameters on the tested observations replaces $F_0$ by the random $F_{\hat{\theta }}$. The resulting transformed observations need not be independent and identically distributed uniforms, so the ordinary reference law fails. The Lilliefors test addresses a particular fitted normal distribution. More general parametric bootstrap calibration must simulate from the fitted statistical model and repeat the fitting in every replicate; holding the fitted statistical parameters fixed during those comparisons calibrates a different procedure. Such a parametric bootstrap requires its own validity justification rather than inheriting exactness from the simple-null test.

For a discrete probability distribution, $F_0(X)$ is generally not uniform. Equal observations also arise from rounding, so tied data require a calibration that accounts for the discrete probability distribution or the rounding rule. The discrepancy remains meaningful, but continuous-null p-values are not automatically valid. Statistical dependence likewise changes the reference law. The two-sample Kolmogorov-Smirnov test compares two empirical distribution functions and has a different calibration.^[ref]

A mixed probability distribution combines a discrete probability distribution with a continuous probability distribution. For example, put probability $1/4$ at zero and probability $3/4$ into a uniform distribution on $(0,1)$. Its cumulative distribution function is $0$ for $x<0$, $1/4+3x/4$ for $0\le x<1$, and $1$ for $x\ge 1$. The jump at zero and the continuous increase both contribute; neither an ordinary probability density function alone nor a probability mass function alone describes the whole law.

Use a fixed logarithm base $b>1$, and write $c={\log }_{b}e=1/\ln b$. All relative entropies below use this same base. Adopt $0\log (0/q)=0$ and $p\log (p/0)=+\infty$ for $p>0$.

For Gibbs inequality, if $q(x)=0<p(x)$ at some point, the relative entropy is infinite and the required conclusion is immediate. Otherwise let $S=\{x:p(x)>0\}$. The elementary inequality $\ln u\le u-1$, with equality only at $u=1$, gives, for every $x\in S$,ConsequentlyThis argument applies to a countable discrete probability distribution too: the difference between the two sides of the pointwise inequality is nonnegative, and the negative part of the entropy summand is bounded by $cq(x)$, hence summable. Thus no subtraction of divergent series is involved. Equality forces both ${\sum }_{S}q=1$ and equality in every pointwise inequality; hence $q(x)=p(x)$ on $S$ and $q=0=p$ off $S$. Conversely $p=q$ plainly gives zero. Therefore $\boxed{D(p\Vert q)\ge 0,D(p\Vert q)=0⟺p=q}$.

For the log-sum inequality, set $F={\sum }_{A}f$, $G={\sum }_{A}g$, and normalize to the probability distributions $p_A=f/F$, $q_A=g/G$. When $A$ is nonempty, positivity ensures $F,G>0$, and direct expansion givesThus normalization reduces the log-sum inequality to Gibbs inequality. Equality holds exactly when $f/F=g/G$. The empty-set case is the zero identity under the usual zero-mass convention.

For the final bound, partition the alphabet using $A=\{x:p(x)>q(x)\}$ and put $a=p(A)$, $d=q(A)$. Since the total signed difference is zero,Apply the log-sum inequality separately to $A$ and its complement. The same normalization argument works for countable sets of finite total mass, using the countable Gibbs inequality just proved. It yields the binary partition bound for relative entropyZero block masses are interpreted by the stated conventions; a positive mass divided by zero makes the bound infinite. The given binary estimate now impliesThis is Pinsker's inequality. Equivalently, for the total variation distance $\Vert p-q{\Vert }_{\mathrm{TV}}=\frac{1}{2}{\sum }_x|p(x)-q(x)|$, the bound is $D(p\Vert q)\ge 2{\log }_{b}e\Vert p-q{\Vert }_{\mathrm{TV}}^2$. The factor ${\log }_{b}e$ keeps the statement valid for either bits or natural-logarithm units.

Take a discrete probability distribution on three points:The probabilities are positive and sum to one. Direct calculation givesandNevertheless it is not a symmetric distribution, since $\mathbb{P}(Y=3)=1/10$ while $\mathbb{P}(Y=-3)=0$. Vanishing of just the first and third moments does not force all odd moments to vanish.

The probability mass at an outcome $x$ is the probability of its singleton, $\mathbb{P}(X=x)$. It equals the jump $F_X(x)-F_X(x^{-})$ of the cumulative distribution function. For a discrete probability distribution these masses form the probability mass function; a continuous probability distribution gives every individual outcome mass zero. Positive probability mass at a point is distinct from the height of a probability density function.

For a discrete probability distribution, the probability mass function of its random variable $X$ isThe sum ranges over its finite or countable set of possible values. An event's probability is the sum of the masses of the included values. For the example under probability distribution, $\mathbb{P}(X\ge 1)=p_X(1)+p_X(3)=0.8$. The cumulative distribution function sums masses below a threshold; its jump at a value equals that value's probability mass. This differs from a probability density function, whose height is not a point probability.

A return level answers the inverse question to a return period: which threshold has the specified rarity? Let $F(z)=F_{\mathrm{annual}}(z)=\mathbb{P}(M_j\le z)$ be the common cumulative distribution function of annual sample maxima in a model with stationarity. For a continuous, strictly increasing $F$ and $T>1$, the $T$-year return level isIt has the same units as $M_j$, whereas $T$ is a count of annual blocks. Independence between years is unnecessary for this one-year quantile definition; it is needed for the geometric waiting and multi-year formulas under return period. For a discrete probability distribution, use the generalized quantile function $F^{-1}(q)=\inf \{z:F(z)\ge q\}$. Then $\mathbb{P}(M_j>z_T)\le 1/T$, with equality not guaranteed at jumps; an exact requested exceedance probability may be unattainable.

For a concrete standard model, choose the maxima convention of the Gumbel distribution, with location $\mu$ and scale $b>0$:The second formula follows by taking logarithms twice and solving $F(z_T)=1-1/T$. The maxima cumulative distribution function and its inverse are given in NIST: extreme value type I distribution, maximum case. With the deliberately chosen toy values $\mu =10$ and $b=2$ in arbitrary units, $z_{10}=14.501$, $z_{100}=19.200$, and $z_{1000}=23.815$, rounded to three decimals. In particular, $19.200$ is approximately the annual quantile at cumulative probability $0.99$; it is not a prediction of the largest observation in the next 100 years. Under independence, the probability that all those annual sample maxima stay below the exact $z_{100}$ is $0.99^{100}\approx 0.366$.

The time block matters. If a toy year contains $m$ independent daily blocks, each with exceedance probability $q$ at one fixed threshold, its annual exceedance probability is $p=1-(1-q{)}^m$, giving annual return period $1/[1-(1-q{)}^m]$. The reciprocal $1/q$ counts daily blocks. Dependence among daily exceedances requires their joint law instead; a threshold crossed on several successive days still produces only one exceedance indicator for that year's sample maximum. The peaks-over-threshold method and declustering of extremes address a different choice of extreme-event sampling.