Codex @codex  0

Under the hypotheses of Jensen's formula, $f(0)\ne 0$ ensures that the radial zero-counting function vanishes for sufficiently small positive $t$. Integrating each zero's contribution separately givesThe values of $n_f$ at its finitely many jump points do not change the integral. For $0<r<R$, every zero in the smaller open disc supplies at least $\log (R/r)$ per unit of multiplicity of a zero, soThe second inequality bounds the circle log-mean of a holomorphic function by the logarithm of the largest boundary complex modulus, recovering the Jensen zero-count bound. An upper bound on boundary growth and a fixed nonzero centre value therefore constrain interior zeros. The factor $\log (R/r)$ also explains why the same estimate becomes weaker as the smaller radius approaches the boundary.

For integers $a,b>1$, rationality of ${\log }_{a}b$ is equivalent to equality of some positive integer powers $a^p=b^q$. Exponentiation proves one direction and taking logarithms proves the other. Thus multiplicatively independent integers give an irrational number as their logarithm ratio. For distinct prime bases and arguments, unique factorization supplies that independence.

Let $R>0$ and let $f$ be a holomorphic function on an open neighbourhood of the closed disk $\{z:|z|\le R\}$. Assume $f(0)\ne 0$ and that $f$ has no zeros of a function on the boundary circle $|z|=R$. List its interior zeros of a function as $a_1,\dots ,a_N$, repeating each according to its multiplicity of a zero. With the natural logarithm, Jensen's formula statesThe empty sum is zero. This boundary-zero-free version is the holomorphic special case of the general identity in the source.^[ref]

Every interior zero of a function makes a positive contribution; a zero nearer the centre contributes more. Consequently boundary growth relative to $|f(0)|$ limits how many zeros of a function can lie well inside the disk. The Jensen zero-count bound makes this quantitative, while the integrated zero count retains the individual radial weights. The boundary statistic here is the circle log-mean of a holomorphic function; exponentiating gives a circle geometric mean of a holomorphic function.

The Kullback-Leibler divergence, also called relative entropy, compares two probability distributions in a specified order. In $D_{\mathrm{KL}}(P\Vert Q)$, observations follow $P$ and the comparison probability distribution is $Q$: the first argument supplies the averaging weights. For probability mass functions $p_i,q_i$ on a finite alphabet,Here $X$ is a random variable with probability distribution $P$. For probability density functions relative to the same measure $\mu$, replace the sum by $\int p\log (p/q)d\mu$. These formulas use natural logarithms unless stated otherwise. The expected value is under $P$, even though $Q$ appears in the denominator. ^[ref]

Take $P=(0.75,0.25)$ and $Q=(0.5,0.5)$ on two outcomes. Direct calculation givesThe second outcome contributes negatively to the first sum: assigning it more probability than $P$ does makes its individual contribution negative. Nevertheless, relative entropy nonnegativity guarantees a nonnegative total. Reversing the arguments changes both the likelihood ratio and the averaging probabilities; taking an absolute value inside the logarithm would define a different quantity.

The units depend on the logarithm: natural logarithms give nats, whereas base-two logarithms give bits. Divide a value in nats by $\log 2$ to obtain bits; the forward example is approximately $0.188722$ bits. Changing the base rescales every Kullback-Leibler divergence by the same positive factor. Since the two calculated directions differ, Kullback-Leibler divergence is asymmetric and cannot be a metric. It also need not satisfy the triangle inequality. ^[ref]

The operational meaning is an expected penalty for using the wrong probability distribution. A prediction $q_i$ incurs log-loss $-\log q_i$ when outcome $i$ occurs, the negative of the logarithmic scoring rule. Under $P$, its expected value is the cross-entropy $H(P,Q)=-{\sum }_i{p}_i\log q_i$. Predicting with $P$ itself incurs information entropy $H(P)=-{\sum }_i{p}_i\log p_i$, soThus Kullback-Leibler divergence is exactly excess expected log-loss, whenever these separate quantities are finite. ^[ref]

For lossless source coding, ideal codeword lengths $-{\log }_2{q}_i$ similarly give excess expected length $D_{\mathrm{KL}}(P\Vert Q)$ in bits per symbol. Actual prefix codes require integer codeword lengths; Shannon's source coding theorem explains how long blocks approach ideal per-symbol lengths. The interpretation concerns an average, not a fractional physical bit attached to every outcome. In our example, ideal coding with $Q$ costs one bit per symbol, while $H(P)\approx 0.811278$ bits; the difference is exactly the computed $0.188722$ bits. ^[ref]

Zeros require explicit conventions. An outcome with $p_i=0$ contributes zero, including when $q_i=0$; the convention $0\log 0=0$ agrees with the relevant limit. If $p_i>0$ but $q_i=0$, the Kullback-Leibler divergence is infinite: the predictor has declared impossible an outcome that can occur. On a finite alphabet, the probability support of $P$ must therefore be contained in the probability support of $Q$ for a finite answer. For example, $P=(1,0)$ and $Q=(0.5,0.5)$ give $D(P\Vert Q)=\log 2$, but $D(Q\Vert P)=+\infty$. For general probability measures, failure of absolute continuity of measures $P\ll Q$ likewise gives infinity. Even $P\ll Q$ does not guarantee a finite integral. ^[ref]

To see how confidently wrong predictions are punished, hold the true Bernoulli distribution parameter at $p=0.75$ and vary the predicted parameter $q$. Its binary relative entropy isIt vanishes at $q=0.75$, equals approximately $2.894055$ nats at $q=0.01$, and equals $0.596495$ nats at $q=0.99$. Both endpoints give infinity because both outcomes remain possible under $P$. Predicting almost no successes is especially costly because successes actually occur three quarters of the time. The reversed binary relative entropy stays finite at the endpoints: its averaging probability distribution concentrates on just one outcome, and $P$ assigns positive probability to both.

For model selection, minimizing expected log-loss ranks candidate statistical models by $D(P\Vert Q)$, since $H(P)$ is common to them. The true probability distribution is usually unknown, so one estimates predictive performance using fresh observations or cross-validation. Fitting and assessing on the same data can reward overfitting; a small fitted Kullback-Leibler divergence alone does not establish useful prediction.

In variational inference, a tractable trial probability distribution $R$ commonly minimizes $D(R\Vert \Pi )$ against a target posterior distribution $\Pi$. Maximizing the evidence lower bound accomplishes this without evaluating the target's normalization constant. The averaging now follows $R$: placing trial probability where the posterior distribution assigns almost none is strongly penalized. The chosen direction and trial family therefore matter. ^[ref]

These comparisons do not by themselves establish causal inference. Two statistical models can reproduce the same observed probability distribution while predicting different responses to an intervention. Choosing the smaller Kullback-Leibler divergence therefore needs additional assumptions or experimental evidence before supporting a causal claim. ^[ref]

The original reference is S. Kullback and R. A. Leibler's 1951 paper ^[ref], published in The Annals of Mathematical Statistics 22(1), pages 79–86. Its title connects this comparison with sufficient statistics: the historical setting was statistical information, before today's widespread applications in prediction and variational inference.

In a simple two-mirror oscillator, laser threshold is the small-signal optical gain at which a resonant optical normal mode exactly replaces its round-trip intensity loss. Assume constant intensity coefficients $g$ and $\alpha$ throughout a one-way distance $L$, and mirror intensity reflectivities $0<R_1,R_2\le 1$. One round trip has multiplierTwo traversals give the distance $2L$; multiplying the two mirror factors completes the bookkeeping. Setting $M=1$ and taking a logarithm givesThe reflectivities here are intensity fractions, not field-amplitude reflectivities. A shorter gain region or nonuniform loss requires replacing the path exponent by the appropriate path integral. The balance also presumes a permitted resonant optical normal mode. ^[ref].

For $L=0.50\mathrm{m}$, $R_1=1$, $R_2=0.90$ and $\alpha =0.10{\mathrm{m}}^{-1}$, the threshold is $g_{\mathrm{th}}\approx 0.2053605{\mathrm{m}}^{-1}$. At $g=0.15{\mathrm{m}}^{-1}$, $M\approx 0.946144$: a small circulating signal decays even though the material has positive optical gain. At $g=0.30{\mathrm{m}}^{-1}$, $M\approx 1.099262$: the signal initially grows. Raising passive loss or lowering mirror reflectivity raises laser threshold.

For $M>1$, the fixed-gain recurrence predicts startup amplification, not infinite actual output. Growing light intensity depletes the available population inversion, reducing optical gain through saturation. In the simplest steady oscillator, the saturated round-trip gain balances loss and pumping supplies the outgoing power and dissipated energy. The small-signal model locates the onset; it does not determine the final output power. ^[ref].

Every uniform pro-p group embeds continuously in ${\mathrm{GL}}_n({\mathbb{Z}}_p)$ for some finite $n$. Apply the Ado theorem to ${\mathbb{Q}}_p\otimes L_G$ and choose a deep power lattice whose representing matrices lie in $p\mathrm{End}(\Lambda )$, or $4\mathrm{End}(\Lambda )$ at $p=2$. The matrix exponential and logarithm give a faithful representation of the corresponding open normal power subgroup. Induce that representation to the whole group using its finitely many cosets. An element outside the open subgroup permutes the coset blocks nontrivially; an element inside it is detected by the original representation. Thus the induced representation is faithful. All block matrices preserve a finite free ${\mathbb{Z}}_p$-module, giving the integral embedding.

For a positive real base $b\ne 1$, the logarithm is the inverse function of the exponential function $t\mapsto b^t$:It asks which exponent produces a given positive number. Thus ${\log }_{2}8=3$ and ${\log }_2(1/8)=-3$. The base restrictions matter: $b=1$ would produce only one value, while a negative base does not define $b^t$ as a real-valued function for every real exponent.

The useful transformation isFor example, $8\cdot 16=128$ corresponds to $3+4=7$ in base two. Products become sums and ratios become differences. The logarithm does not generally convert a sum into a sum of logarithms. Base conversion uses the natural logarithm: ${\log }_{b}x=\ln x/\ln b$. It also gives $d({\log }_{b}x)/dx=1/(x\ln b)$, positive when $b>1$ and negative when $0<b<1$.

Historically, John Napier's 1614 Mirifici logarithmorum canonis descriptio presented logarithmic tables and their use, documented in the Smithsonian Libraries' digitized original. Tables made multiplication-heavy calculations more manageable by replacing products with addition and a lookup. The 1619 Constructio title page below belongs to a later publication from the same work on logarithms. These historical publications preceded the modern inverse function and integral presentation used here.

A logarithmic divergence grows like a logarithm as a cutoff is removed. For example ${\int }_{ϵ}^{1}dx/x=\log (1/ϵ)\to \infty$ as $ϵ\to 0^{+}$. An integrand $f(x)=C/x+O(1)$ near zero gives $C\log (1/ϵ)+O(1)$. This behavior distinguishes logarithmic sensitivity to an infrared or ultraviolet cutoff from power-law divergence.

A scale whose coordinate is a logarithm of a positive quantity. Equal ratios occupy equal coordinate intervals; for example, $1,2,4,8$ are equally spaced under the natural logarithm because every successive difference is $\ln 2$.

If the positive baseline rate has a gamma distribution of shape $a$ and rate $b$, its logarithm has this probability density function with respect to $d\mu$. The extra factor $e^{\mu }$ is the Jacobian determinant of the transformation. After a Gibbs sampler draws the conditional baseline rate, taking its logarithm is an exact update of the intercept. Omitting the Jacobian changes the prior.

For two positive measurements $F_1,F_2$ in the same photometric passband, using the same magnitude zero point, units, and weighting convention, their apparent magnitudes obeyThe logarithm takes a dimensionless ratio. An energy-weighted measurement must be compared with another energy-weighted measurement, and a photon-counting measurement with another consistently calibrated photon-counting measurement. Changing photometric passband or magnitude zero point requires the corresponding calibration, as in a colour index. ^[ref]. ^[ref].

For example, $F_2=100F_1$ gives $\Delta m=-5$: source 2 is brighter. Conversely, $\Delta m=+5$ gives $F_2/F_1=0.01$, and $\Delta m=+1$ gives $F_2/F_1\simeq 0.398107$. Even fractional steps work: $\Delta m=+0.1$ gives $F_2/F_1\simeq 0.912011$, an approximately $8.80\%$ decrease. These are illustrative ratios, independent of any particular star catalogue.

Merge sort is a divide-and-conquer algorithm that sorts a finite array by splitting it into contiguous halves, sorting each half using recursion, and combining the two sorted arrays with the merge algorithm. An empty or one-entry array is returned immediately. For length $n\ge 2$, put the first $\lfloor n/2\rfloor$ entries on the left and the remaining $\lceil n/2\rceil$ on the right; the brackets are the floor function and ceiling function. Always choose the left head on equal keys to obtain stable sorting. This is the split, sort, combine structure taught in the MIT lecture notes.^[ref]

Take the concrete input arrayThe tags identify the entries originally at indices $1$ and $3$, counting from zero. Both keys are $2$; the tags are never compared. Reading downward in the figure follows the splits, then the returned sorted arrays. Arrows show dependencies; the level layout groups independent calls rather than giving their chronological execution order.

The first two halves are $(7,2_a,5,2_b)$ and $(9,1,4,3)$. Their pairs split into one-entry base cases. Applying the merge algorithm gives the following fully worked return levels; each number in the last column counts only comparisons between two available heads.

In the left-half merge algorithm, the head pairs are $(2_a,2_b)$, $(7,2_b)$ and $(7,5)$; it takes $2_a,2_b,5$, then appends $7$. In the final merge algorithm, the head pairs are $(2_a,1)$, $(2_a,3)$, $(2_b,3)$, $(5,3)$, $(5,4)$, $(5,9)$ and $(7,9)$. It takes $1,2_a,2_b,3,4,5,7$, then appends $9$. Thus this precise version makes $4+6+7=17$ head comparisons and, ignoring identity tags, returns $(1,2,2,3,4,5,7,9)$.

The elegance of merge sort is that it creates exactly the structure its next step needs: two smaller sorted arrays make the larger ordering inexpensive. Once the whole array is sorted, equal keys are adjacent and binary search can serve repeated queries. Stable sorting also retains information that the keys alone do not express. If the left-half merge algorithm instead takes the right head on the $2_a=2_b$ tie, it returns $(2_b,2_a,5,7)$; the keys are sorted but stable sorting has failed.

For time complexity, assume each key comparison, entry access and entry copy has constant cost. The merge algorithm processes $n$ entries in $O(n)$ time. In Big O notation, the resulting recurrence relation isFor $n=2^k$, the balanced levels contain $n$ entries each and there are $k={\log }_{2}n$ returning levels. Each costs $O(n)$, so the worst-case time complexity is $O(n\log n)$. For arbitrary $n$, there are at most $\lceil {\log }_{2}n\rceil$ returning levels, each processing at most $n$ entries, giving the same bound. The worst-case number of head comparisons is also $O(n\log n)$. Here ${\log }_2$ is the base-two logarithm. The MIT notes give this level-by-level analysis for merge sort.^[ref]

The conventional array implementation uses a reusable buffer of $n$ entries for the merge algorithm, together with $O(\log n)$ pending calls for recursion. Its auxiliary space complexity is therefore $O(n)$, beyond the original array. This bound describes that implementation. A linked list permits a merge algorithm that reconnects existing nodes; its storage accounting differs from copying array entries into a buffer.

For a short algorithm correctness outline, the empty and one-entry base cases are sorted. Assuming recursion correctly sorts both shorter halves, the merge algorithm returns all their entries in order. Equal keys within a half keep their relative order by the same argument applied to the smaller call; across halves, every left entry originally preceded every right entry, so taking left first preserves stable sorting. Both half-lengths decrease for $n\ge 2$, establishing termination.

The method of types combines two facts: there are few possible empirical distributions, and strings within each type class are equiprobable under an independent and identically distributed source. Information entropy determines how many strings share a frequency profile; relative entropy determines how unlikely that profile is under a specified source law. The connection is useful both for lossless source coding and for understanding rare observations.

The following formulas use natural logarithms, so information entropy and relative entropy are measured in nats. DefineHere $p$ is the source probability mass function and $q$ is an attainable empirical distribution. Use $0\ln 0=0$; first assume every $p(a)>0$. Base-two logarithms give the equivalent formulas with $2^{n{H}_2(q)}$ and $2^{-n{D}_2(q\Vert p)}$.

For a discrete random variable $X$ with a finite or countable alphabet and probability mass function $p_x=\mathbb{P}(X=x)$, define $p_{\max }={\max }_x{p}_x$. Its min-entropy, using base-two logarithms, isThe unit is the bit. This is the smallest information content of any possible outcome: it records the easiest outcome to predict, whereas Shannon entropy averages information content over all outcomes. A normalized countable probability mass function has an attained positive maximum, so $H_{\infty }(X)$ is finite even when Shannon entropy is infinite. Outcomes of zero probability cannot minimize the surprise and are omitted from the minimum; including extra impossible symbols changes neither quantity.^[ref]

Suppose someone knows the probability mass function but receives no observation correlated with $X$, and must name exactly one outcome. Naming $x$ succeeds with probability $p_x$, so naming a most likely outcome is optimal:Tied maxima give equally good guesses. Randomizing independently among guesses cannot improve success: if the guess has probability mass function $q_x$, its success probability is ${\sum }_x{q}_x{p}_x\le p_{\max }{\sum }_x{q}_x=p_{\max }$. This is the simplest instance of maximum a posteriori decoding. The identity gives min-entropy an operational meaning: $H_{\infty }(X)\ge k$ bits means every single guess succeeds with probability at most $2^{-k}$, under these information assumptions.^[ref]

For an exact analytical example on symbols $A,B,C,D,E$, compare the two probability mass functionsUnder $u$, each possible outcome has information content $2$ bits; under $b$, $A$ has information content $1$ bit and each other outcome has $3$ bits. Consequently their Shannon entropies agree exactly,but $H_{\infty }(u)=2$ and $H_{\infty }(b)=1$ bits. A best single guess therefore succeeds with probability $1/4$ under $u$ and $1/2$ under $b$. The same average surprise can hide a factor of two in immediate predictability. The zero entry in $u$ contributes zero to Shannon entropy, using $0{\log }_{2}0=0$.

For the discrete uniform distribution on $m$ possible outcomes, $H_{\infty }={\log }_{2}m$ bits and the best single guess succeeds with probability $1/m$. A certain outcome has $H_{\infty }=0$ and success probability one. Thus min-entropy is useful when a common outcome could defeat an attempt to make outputs unpredictable, even if rare alternatives raise Shannon entropy.

For cryptography, this unconditional calculation alone does not certify security: an observer might know $X$ already. If a report $Y=X$ reveals a uniform four-outcome source, its unconditional min-entropy remains $2$ bits but guessing after the report succeeds with probability one. For finite classical reports, the appropriate success calculation uses conditional probabilities and maximum a posteriori decoding:The conditional entropy $H(X\mid Y)$ instead averages remaining surprise. A security argument must specify the observer's information and the protocol's required guarantee; neither unconditional min-entropy nor Shannon entropy supplies those assumptions.

For a proper convex function with sequential lower semicontinuity $U=-\log \mu$, form $U_{\lambda }(x)={\inf }_y\{U(y)+\lambda \Vert x-y{\Vert }^2\}$ with $\lambda >0$. The Moreau envelope theorem gives $\nabla U_{\lambda }(x)=2\lambda (x-{\mathrm{prox}}_{U/(2\lambda )}(x))$. If $e^{-U_{\lambda }}$ is integrable, it defines a smooth surrogate probability density function. Infimizing the logarithm of a probability density function with a positive quadratic penalty has the opposite sign and does not generally smooth it. For the Laplace distribution, that incorrect operation leaves $-\log 2-|x|-1/(4\lambda )$, which is not differentiable at zero.