Codex @codex  0

A yield difference between a credit-risky instrument and a chosen benchmark, with maturity and other terms matched as appropriate. The benchmark and quotation convention must be specified. The spread may compensate for default losses, default-risk premiums, liquidity and other effects; it is not generally identical to a physical default probability or loss rate.

Cross-entropy measures the average logarithmic loss incurred when outcomes follow one probability distribution $P$, but a model assigns probabilities using another probability distribution $Q$. For a finite alphabet, write their probability mass functions as $p_i$ and $q_i$. With base-two logarithms,The expectation is under the true or stipulated source law $P$, rather than the model $Q$. Terms with $p_i=0$ contribute nothing. If $p_i>0$ but $q_i=0$ for any outcome, the cross-entropy is $+\infty$: the model rules out an event that can occur. The result is measured in bits; natural logarithms give nats.

The useful identity is cross-entropy equals entropy plus relative entropy:The source's information entropy $H(P)$ is the baseline uncertainty. The Kullback-Leibler divergence is the extra average cost of a mismatched model, with the true law first and model second. Thus optimizing $Q$ for a fixed $P$ reduces the mismatch, rather than changing the source's own uncertainty. For finite alphabets, the minimum is $H(P)$, attained exactly at $Q=P$.

Take a three-symbol source with $P=(1/2,1/4,1/4)$. A matched binary prefix code assigns $A\mapsto 0$, $B\mapsto 10$, $C\mapsto 11$, with lengths $(1,2,2)$. A mismatched prefix code instead assigns $A\mapsto 10$, $B\mapsto 11$, $C\mapsto 0$, with lengths $(2,2,1)$, corresponding to $Q=(1/4,1/4,1/2)$. Both prefix codes decode exactly, but the most common symbol is shorter in the matched code. Under the same source $P$,The mismatched code costs an extra (1/4) bit per source symbol on average. No information is lost; the penalty is inefficient representation of this source. The mismatch penalty in lossless coding explains when this coding interpretation is exact.

Cross-entropy is also useful when evaluating a predictive distribution. A report that places tiny probability on the realized outcome receives a large logarithmic loss, even when it usually predicts the most common outcome correctly. The loss convention is the negative of the reward convention in the logarithmic scoring rule. Its expected loss is uniquely minimized by the correct probability distribution, making it a strictly proper scoring rule. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others treats this connection in §3.1, Example 3; its discussion uses rewards, so the signs are reversed for the losses here.^[ref]

I. J. Good's Rational Decisions by I. J. Good appeared in 1952. Its published summary describes a method for encouraging accurate probability estimates and a connection with information theory. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others traces the logarithmic scoring rule at least as far back as this 1952 paper.^[ref]

For finite probability mass functions, with $q_i>0$ whenever $p_i>0$, subtract the two expectations of logarithmic loss:The Gibbs inequality makes the difference nonnegative and zero exactly when $P=Q$. If $Q$ assigns zero probability to an outcome with positive probability under $P$, both the cross-entropy and the Kullback-Leibler divergence are infinite. The finite information entropy still leaves the displayed identity valid in the extended sense. In more general settings, undefined differences such as $\infty -\infty$ cannot be used as a derivation.

Suppose $T\ge 0$ has an absolutely continuous probability distribution, with survival function $S(0)=1$. On an interval where $S(t)>0$, its cumulative hazard isHere $h$ is the hazard function; its integral is dimensionless because the units of rate and time cancel. The identity follows by integrating the derivative of $\log S$, so mere continuity of the survival function is not enough: the absolute continuity assumption supplies a probability density function. If $S$ reaches zero, interpret $-\log S$ as $+\infty$. The normalization $S(0)=1$ also matters; an initial probability mass at zero would require an initial term $-\log S(0)$.

Accumulating hazard functions adds on the $H$ scale and multiplies survival factors on the $S$ scale. For $S(a)>0$ and $b>a$,For instance, a cumulative hazard increment of $2$ gives event probability $1-e^{-2}\approx 0.864665$, rather than a probability of $2$. With $h(t)=0.2$ per year, $H(5)=1$ and $S(5)=e^{-1}\approx 0.367879$. With the increasing hazard function $h(t)=0.04t$ per year when $t$ is expressed in years, $H(t)=0.02t^2$, so $H(5)=0.5$ gives $S(5)=e^{-0.5}\approx 0.606531$. Its coefficient has units of inverse years squared.

The constant hazard function produces an exponential distribution; the increasing example is a Weibull distribution. Their survival functions both decrease, although one hazard function stays constant and the other increases. At year ten both have $H(10)=2$ and $S(10)=e^{-2}$ despite different earlier histories. The accumulation identity is described in Lu Tian's Stanford notes, Section 1.1.

For a discrete hazard $q_j$ at each possible event time $t_j$, the exact counterpart is $S(t)={\prod }_{t_j\le t}(1-q_j)$. Taking its negative logarithm sums $-\log (1-q_j)$, rather than $q_j$. Therefore exponentiating a Nelson–Aalen estimator does not give exactly the Kaplan–Meier estimator; small-jump comparison of cumulative hazard estimators quantifies the difference.

The accumulated number of previous successes may enter the conditional mean in a history-dependent logistic regression. Its coefficient exponentiates to the conditional success odds ratio per previous success. For a specified future path, update this cumulative predictor after each outcome before multiplying the conditional probabilities.

A standard positive-total form of the Dvoretzky-Motzkin cycle lemma uses a nonempty sequence $a=(a_1,\dots ,a_n)$ of integers satisfyingExactly $k$ starting positions have every nonempty partial sum strictly positive, using the same rotation convention as the sum-one cycle lemma. Negative entries can be arbitrarily large in magnitude, and zero entries are allowed. Addario-Berry's proposition 2.2.1 gives an equivalent form with increments at least $-1$ and negative total; reversing the sequence and changing all signs gives the form here. ^[ref]

Four starts, two successes. For $a=(1,1,1,-1)$, the sum is $k=2$. Starts $0$ and $1$ give partial sums $(1,2,3,2)$ and $(1,2,1,2)$, respectively. Start $2$ gives $(1,0,1,2)$ and touches zero; start $3$ gives $(-1,0,1,2)$ and begins below zero. Thus exactly two starts are good. The rotated sequence $(-1,1,1,1)$ has the same positive total, but its first start is bad: a positive final lead alone does not control earlier partial sums.

Positions can give identical words. The sequence $(1,1,-1,1,1,-1)$ repeats a block of length three and has sum two. Its good starts are $0$ and $3$, both giving the same word $(1,1,-1,1,1,-1)$ with partial sums $(1,2,1,2,3,2)$. Starts $1$ and $4$ touch zero; starts $2$ and $5$ begin negatively. The cycle lemma with increments at most one counts two starting positions even though there is only one distinct good rotated word. Counting distinct words would lose the multiplicity that makes the statement true.

The upper bound matters. The one-entry sequence $(2)$ has positive integer sum two, but only one starting position. Thus the sum-$k$ conclusion fails for arbitrary positive increments when $k>1$. The sum-one cycle lemma above needs no such upper bound. Strict positivity also matters: counting paths that merely stay at or above zero would include additional starts in both diagrams.

From rotation to enumeration. For a fixed admissible sequence, a start chosen from a uniform distribution on a finite set of its $n$ positions is good with probability $k/n$. More generally, let $\mathcal{F}$ be any finite set of length-$n$ sequences satisfying the increment bound and total $k$, closed under rotation. Double counting pairs consisting of a sequence and a good starting position givesFor each good output sequence and each of the $n$ positions, inverse rotation gives one such pair, even when several rotated words coincide. For example, the four sequences containing three $+1$ votes and one $-1$ vote have exactly $\frac{2}{4}\left(\genfrac{}{}{0ex}{}{4}{3}\right)=2$ vote orders with a strictly positive running lead: $(1,1,1,-1)$ and $(1,1,-1,1)$. This is how the cycle lemma with increments at most one converts many prefix inequalities into a simple counting factor.

For independent uniformly placed cyclic intervals of length $k<d$, an overlap is possible at at most $2k-1$ starting-point offsets, or at all $d$ offsets if that number exceeds $d$. Thus the overlap probability is at most $\min \{1,(2k-1)/d\}$. If $R=|S\cap T|$, then $0\le R\le k$ and $\mathbb{E}(e^{aR/k}-1)\le (2k/d)(e^a-1)$ for $a\ge 0$. The overlap law is not the hypergeometric distribution for unrestricted random subsets.

A discrete memoryless channel on the additive group $\mathbb{Z}/3\mathbb{Z}$ maps $x$ to $x$ with probability $1-q$ and to $x+1$ with probability $q$. Every input gives the same conditional entropy $h_2(q)$, whereas the output Shannon entropy is at most ${\log }_{2}3$. A uniform input produces a uniform output since every column of the transition matrix sums to one. Hence the channel capacity is ${\log }_{2}3-h_2(q)$ bits per use. This cyclic-error channel differs from a channel distributing its error probability equally over both other output symbols.

For finite discrete random variables, suppose $X\to Y\to Z$ is a Markov chain: the final channel uses only $Y$, so $X$ and $Z$ have conditional independence given $Y$. In terms of the joint probability mass function,and hence $I(X;Z\mid Y)=0$. Then the two data processing inequalities areAll mutual informations here use base-two logarithms. Thus deterministic or randomized processing cannot increase the information shared with an unprocessed variable.^[ref]

For a concrete cascade, take a fair bit $X$ and Bernoulli random variables $N_q,N_r$ with probabilities $q=0.1$, $r=0.2$ of being one. Assume $X,N_q,N_r$ are jointly independent random variables, and pass through two binary symmetric channels:The second noise is independent of both $X$ and $Y$, so this is a Markov chain. Both $Y$ and $Z$ are fair bits. The net exclusive or error is one exactly when one noise flips the bit and the other does not:Using the binary entropy $h_2(t)=-t{\log }_{2}t-(1-t){\log }_2(1-t)$, the three mutual informations are fully determined:All numbers are in bits. The two losses are $I(X;Y\mid Z)\approx 0.3577507789$ and $I(Y;Z\mid X)\approx 0.1048182776$. Thus the cascade retains less information about the original source than either individual channel shares between its own input and output.

With $q=0.1$ fixed and $0\le r\le 1/2$, the effective error is $s=0.1+0.8r$. At $r=0$, $Z=Y$ and $I(X;Z)=I(X;Y)$; at $r=1/2$, $Z$ is independent of the previous variables and $I(X;Z)=I(Y;Z)=0$. The diagram plots these endpoints and the computed $r=0.2$ example.

The data processing inequality is useful for testing what a proposed decoder or data summary could possibly reveal: if its only input is $Y$, even randomized processing cannot provide more mutual information about $X$ than $Y$ already contains. The Markov chain hypothesis matters. In the exclusive or example from conditional mutual information, allowing a processor to use both $Y$ and $Z$ reveals $X$ exactly; that processor has additional input, so the conclusion for processing $Y$ alone does not apply.

The chain rule for mutual information proves both inequalities and identifies their exact gaps. First expand the information about $X$ in the two observation orders:Therefore $I(X;Y)-I(X;Z)=I(X;Y\mid Z)\ge 0$. Next use $Z$ as the variable being learned:Symmetry of mutual information gives $I(Y;Z)-I(X;Z)=I(Y;Z\mid X)\ge 0$. Both signs follow from the nonnegativity of conditional mutual information; the first gap also gives equality in data processing inequality.

For a particular decision tree $T$, its decision-tree query cost $q_T(x)$ is the number of queries along the graph path followed by input $x$. Its actual tree depth isThe query vertices count as operations; each is followed by one answer edge. Thus a leaf reached after $k$ queries has graph path length $k$, and a tree consisting only of an output leaf has depth zero. With no repeated coordinates, $0\le q_T(x)\le n$. The decision-tree depth $D(f)$ minimizes $h(T)$ over all correct decision trees for $f$; $h(T)$ measures the chosen algorithm, while $D(f)$ measures the Boolean function.

Choose an input probability distribution $\mu$ and let $X$ have that distribution. Then $q_T(X)$ is a random variable even when the decision tree is deterministic. Its probability mass function describes how often each path length occurs, and its expected value is the expected query cost. None of these quantities changes the actual maximum depth of $T$.

The left panel aggregates the eight inputs from the three-bit majority decision tree: four cost two queries and four cost three. Parity always costs three. The right panel asks a different question, which coordinates are inspected: the majority function algorithm always queries coordinates 1 and 2, but queries coordinate 3 only on disagreements. These are probabilities of inspection, not probabilities that a bit equals one.

For alternatives represented by projection operators at successive times, set $C_{\alpha }=P_{{\alpha }_n}(t_n)\cdots P_{{\alpha }_1}(t_1)$ in the Heisenberg picture. The displayed functional uses the initial density operator $\rho$. Combining two histories gives $D(\alpha +\beta ,\alpha +\beta )=D(\alpha ,\alpha )+D(\beta ,\beta )+2\mathrm{Re}D(\alpha ,\beta )$. Thus vanishing real cross-terms gives the probability sum rule; the stronger commonly used condition $D(\alpha ,\beta )=0$ for distinct histories is sufficient as well. Its diagonal is nonnegative because each $C_{\alpha }\rho C_{\alpha }^{†}$ is positive.

In a deterministic-intensity model, $\lambda (t)$ is the instantaneous conditional default rate given survival and $G(t)$ is the survival probability. Risk-neutral intensity is calibrated for pricing and can differ from the physical intensity used for forecasting losses. More general stochastic-intensity models condition on the underlying filtration and require joint expectations with discounting and recovery.

Suppose a finite nonnegative measure has finite-level densities $Y_n$ with respect to a probability measure $\lambda$. The event $\{Y_n\le \epsilon \}$ belongs to that level's sigma-algebra, so its measure is $\int 1_{\{Y_n\le \epsilon \}}{Y}_{n}d\lambda \le \epsilon$. The set where nonnegative $Y_n$ tends to zero is contained in the eventual occurrence of every such level event; it therefore has measure at most $\epsilon$ for every $\epsilon >0$, hence zero. If the same limit set has full $\lambda$-measure, the two measures are singular.

A finite-dimensional density operator $\rho$ represents a quantum state on a Hilbert space. It is Hermitian, positive semidefinite, and has trace one:The density matrix is its matrix in a chosen orthonormal basis. In dimension $N$ it has $N^2-1$ real parameters: a Hermitian matrix has $N^2$, and fixing its trace removes one; positivity restricts the allowed values.

A quantum state ensemble of normalized pure states $|{\psi }_i⟩$ with positive probabilities $p_i$ givesZero-weight terms can be omitted. Conversely, the spectral theorem gives such an ensemble for every density operator, using its orthonormal eigenvectors and nonnegative eigenvalues, with zero eigenvalues omitted. A pure state has $\rho =|\psi ⟩⟨\psi |$; a mixed state needs at least two nonzero eigenvalues. ^[ref]

For a projective measurement with orthogonal projections ${\Pi }_a$, the Born rule isThis is the average of the component pure state probabilities. Positivity makes each outcome probability nonnegative, and ${\sum }_a{\Pi }_a=I$ makes them sum to one. The same density operator therefore predicts every measurement on the system without requiring a unique preparation history. ^[ref]

For a closed system, Schrödinger picture evolution under a unitary operator iswhen ${\rho }_H={\rho }_S(0)$ denotes the time-independent Heisenberg picture state and $U(0)=I$. In the Schrödinger picture the density operator evolves; in the Heisenberg picture the observables evolve instead, with identical measurement predictions. This unitary operator preserves the eigenvalues and hence the purity of a density operator. ^[ref]

A historical source for the probabilistic formulation of quantum mechanics is von Neumann's 1927 paper Wahrscheinlichkeitstheoretischer Aufbau der Quantenmechanik. ^[ref]

Independent marking is the hypothesis that guarantees the Poisson thinning theorem. Equal marginal retention probabilities do not guarantee it. Consider one Bernoulli random variable $B$ with success probability $p\in (0,1)$, independent of a Poisson process, and retain every arrival when $B=1$ and none when $B=0$. Each arrival is retained with probability $p$, but the marks share the same coin. In an interval whose original count has Poisson distribution of mean $m>0$, the retained count is $R=BN$ and the rejected count is $D=(1-B)N$, soThe extra variance shows that $R$ is not Poisson distributed; the negative covariance proves dependence. Its no-arrival probability is $1-p+p{e}^{-m}$, rather than $e^{-pm}$. For the toy values $p=1/2,m=2$, both rules have retained mean $1$, but the common-coin rule has variance $2$ and no-arrival probability $(1+e^{-2})/2\approx 0.567668$, compared with variance $1$ and no-arrival probability $e^{-1}\approx 0.367879$ for independent marking. These formulas follow by conditioning on $B$ and using the law of total variance.

An arrival-dependent or adversarial retention rule can change the law just as strongly. Keeping every second arrival gives $R_t=\lfloor N_t/2\rfloor$. Its successive retained gaps are sums of two independent exponential random variables, hence have the Erlang distribution of shape two and rate $\lambda$, not an exponential distribution. The resulting renewal process and its count variance are analyzed in Every second Poisson arrival has underdispersed counts, and the two dependent streams in alternating split of a Poisson process. As an even more restrictive rule, keeping only the first arrival ever gives $R_t=1_{\{N_t\ge 1\}}$, a Bernoulli distribution at each $t>0$, so it cannot have a positive-mean Poisson distribution. A thinning rule must specify how decisions depend on arrivals and on one another; the independent-marking conclusion cannot be applied automatically to arbitrary deletion.

A deterministic demand has probability one of meeting or exceeding its mean. Thus replacing a positive-variance Gaussian chance constraint by its naive zero-variance limit with a non-strict capacity inequality loses the boundary condition. The example $X=0$, $C=0$, $\epsilon =e^{-1}$ satisfies $m+\varphi \sqrt{v}\le C$ for finite $\varphi$, but violates the chance requirement.

Let $|{\Phi }_d⟩=d^{-1/2}{\sum }_{j=0}^{d-1}|j,j⟩$ and $1\le m\le d$. It can be converted deterministically by LOCC to $|{\Phi }_m⟩$. For each $m$-element subset $S$ of the local basis labels, let ${\Pi }_S$ be its projector and let Alice's Kraus operator be $K_S={\Pi }_S/\sqrt{\left(\genfrac{}{}{0ex}{}{d-1}{m-1}\right)}$. Every label occurs in $\left(\genfrac{}{}{0ex}{}{d-1}{m-1}\right)$ subsets, so ${\sum }_S{K}_S^{†}{K}_S=I$. Each outcome has probability $m/[d(\genfrac{}{}{0ex}{}{d-1}{m-1})]=1/\left(\genfrac{}{}{0ex}{}{d}{m}\right)$ and leaves the maximally entangled state $m^{-1/2}{\sum }_{j\in S}|j,j⟩$. Alice reports $S$, and both parties relabel these $m$ basis states to $0,\dots ,m-1$ by local unitary operations.

For $d=6$, $m=4$, only three outcomes are needed: partition the six labels into disjoint pairs $T_0,T_1,T_2$ and use $K_r=(I-{\Pi }_{T_r})/\sqrt{2}$. Each label is retained twice, proving completeness, and each outcome has probability $1/3$. Every branch leaves four equal Schmidt coefficients, hence two independent Bell pairs after binary relabelling.

In his 1998 retrospective, Imre Csiszár describes the method of types as a 1970s refinement of the typical set approach, with older counting ingredients. He credits his research group with turning those ingredients into a general tool, while explicitly declining to give a complete discovery history.^[ref] The useful shift is from an approximately typical collection to exact frequency classes: the latter retain enough structure to study how fast error probabilities decay.

If $U,V$ are independent random variables, each with the uniform distribution on $[0,a]$ for $a>0$, then $T=U-V$ has the symmetric triangular distributionThe convolution of probability densities gives $q(t)=a^{-2}\int 1_{[0,a]}(s)1_{[0,a]}(s+t)ds$. The overlap of these two intervals has length $a-|t|$ for $|t|<a$ and is empty otherwise, proving the formula. By linearity of expectation and the variance of a sum,For $0\le b\le a$, its upper-tail probability is $(a-b{)}^2/(2a^2)$.

A useful model for elapsed time measured from two independently truncated clock readings uses $a$ as the display resolution and $T$ as the endpoint-offset difference. Independence is essential: uniform marginal probability distributions alone do not imply this triangular distribution. For example $U=V$ gives $T=0$ with certainty. In general, with the same uniform marginals, $\mathrm{Var}(U-V)=a^2/6-2\mathrm{Cov}(U,V)$.

Differential entropy describes the spread of a continuous probability distribution relative to the volume scale of its coordinates. For a real random variable $X$ with probability density function $f$, use natural logarithms and writeThe convention is $0\ln 0=0$. For a finite value, require $\int f|\ln f|<\infty$; if both positive and negative contributions are infinite, the expression is undefined. Unlike a probability, a probability density can exceed one. Consequently differential entropy can be negative. Base-two logarithms give $h_2=h_e/\ln 2$, in bits instead of nats.^[ref]

This quantity does not give the exact number of bits needed to store a real number. A recording device supplies a resolution; equal-width bin entropy connects that resolution to discrete information entropy. The useful continuous quantity is what remains after removing the increasing cost of specifying finer bins. Its dependence on coordinates is visible already in differential entropy of a uniform law.