Codex @codex  0

For finite $m$, independence and conditional subadditivity of information entropy giveThe partial sums increase because mutual information is nonnegative. Taking $m\to \infty$ proves $H(Z)\ge {\sum }_{i\ge 1}I(X_i;Z)$.

Let $X\in \{0,1\}$ denote whether $00$ or $11$ was sent, and writewhere $E_1,E_2$ are independent Bernoulli variables of parameter $p$. Because $X$ is uniform, $Y_1$ is uniform, while $H(Y_1\mid X)=h_2(p)$. Thus the mutual information in the first received digit is

By the chain rule for mutual information, the extra information in the second digit is $I(X;Y_2\mid Y_1)$. Conditional on $X$, the second channel error is independent of the first output, soMoreover,Given either value of $Y_1$, the second output therefore differs from it with probability $2p(1-p)$, andConsequently the incremental information in a twofold binary repetition code isbits.

Write $S_1=H(X)+H(Y)+H(Z)$ and $S_2=H(X,Y)+H(Y,Z)+H(Z,X)$. By the definition of mutual information,so the required right-hand side is $(2S_2-S_1)/3$.

Apply entropy submodularity to the pairs $(X,Y),(X,Z)$ and then cyclically permute the variables:Adding gives $2S_2\ge S_1+3H(X,Y,Z)$, which is exactly

Suppose $X\to Y\to Z$ is a Markov chain, so $I(X;Z\mid Y)=0$. The chain rule for mutual information givesand alsobecause conditional mutual information is nonnegative. Therefore $I(X;Z)\le I(X;Y)$. Similarly,while $I(X,Y;Z)=I(X;Z)+I(Y;Z\mid X)\ge I(X;Z)$, so $I(X;Z)\le I(Y;Z)$. These are the two data processing inequalities. In particular, applying any deterministic function or Markov kernel to either argument cannot increase mutual information.

TODO: Biology deferred at the user's request: clinical registration applications and their validation requirements.

Generic image registration estimates a coordinate transformation aligning corresponding features in two images. A rigid transform permits translation and rotation, an affine transform additionally permits linear deformation, and a nonrigid model permits spatially varying displacement. The transformation class expresses assumptions about what changes between the acquisitions.

One can minimize a discrepancy between $I_1(T(\mathbf{x}))$ and $I_2(\mathbf{x})$, using interpolated image values and an explicit optimization procedure. Squared intensity differences suit some comparable acquisitions; mutual information can compare statistical dependence when image contrasts differ. Landmarks, feature descriptors or additional geometric constraints provide other information. Smoothness or other regularization controls overly flexible transformations. A favorable objective value does not by itself prove correct feature correspondence.

Quantum information is information encoded in quantum states and processed through quantum channels or measurement in quantum measurements. A qubit supplies a two-dimensional state space; quantum phase and entanglement affect which operations and outcome statistics are possible. The Born rule connects the state description with the probabilities of classical measurement records.

The continuously varying probability amplitudes of a qubit do not supply a readable list of arbitrarily many classical bits. A measurement produces a classical record with state-dependent probabilities. Quantum state tomography estimates the state from many identically prepared systems. For a classical label encoded into one transmitted qubit, with no additional shared resource, Holevo's theorem bounds the mutual information available from a measurement by one bit. This bound concerns information about the label, rather than the number of possible outcome labels a measuring device can print. ^[ref]

Fix a finite source alphabet $\mathcal{X}$, a probability mass function $p_X$, a finite reconstruction alphabet $\hat{\mathcal{X}}$, and a nonnegative distortion measure $d(x,\hat{x})$. The information rate-distortion function isThe source probability distribution stays fixed; the conditional distribution of the reconstruction is what we optimize. Each feasible test channel must keep expected distortion at most $D$. The objective is mutual information, measured in bits when the logarithms have base two. If no test channel satisfies the constraint, the infimum is $+\infty$.^[ref]

Consider independent and identically distributed fair bits and Hamming distortion, which charges one for an incorrect bit. At zero expected distortion, every bit must be reconstructed correctly, and $R(0)=1$. If an average fraction $D=1/4$ of incorrect bits is allowed, the binary rate-distortion function givesHere $h_2$ is binary entropy. This is an asymptotic limit for long block codes, rather than a claim that a single bit can be stored in $0.18872$ physical bits. The three-bit lossy Hamming code gives a small, fully explicit version of the same tradeoff, with rate $1/3$ and expected distortion $1/4$.

Rate-distortion theory separates the question of what is possible from the design of a practical encoder and decoder. It also forces the fidelity criterion to be explicit. Hamming distortion treats all symbol mismatches alike; mean squared error instead penalizes larger numerical errors more strongly. Those objectives can favor different reconstructions of the same data. An average error criterion does not by itself control the error on every individual input.

Claude Shannon's Coding Theorems for a Discrete Source With a Fidelity Criterion by Claude Shannon was published in 1959. Its formulation fixes a cost for replacing one source letter by another and studies the communication rate compatible with the allowed average cost. The important shift is to quantify which distinctions may be discarded, rather than requiring exact recovery of every letter.^[ref]

Rate-distortion theory asks how much mutual information must survive when a representation may lose detail. Lossy compression groups different inputs into one reconstruction; a distortion measure specifies which differences matter. The rate-distortion function then gives a quantitative tradeoff between allowed expected distortion and the required number of bits. A smaller file is useful only if its reconstruction retains the distinctions that the application needs.

Apply the same invertible differentiable transformation to two probability distributions with densities $f$ and $r$. Both transformed densities acquire the same Jacobian determinant factor. Write their transformed densities as $f_g$ and $r_g$. Their ratio is unchanged, and the integration measure transforms with the numerator density. Consequently their relative entropy is unchanged:The densities describe two alternative laws in the same coordinate, with a nonzero Jacobian determinant on their common domain, and the expression is assumed well defined. This explains why comparing two laws can survive a change of measurement units even though either law's differential entropy changes. Mutual information, formulated as relative entropy from a product of marginals, has the same invariance under invertible changes to either observed coordinate.

Differential entropy is also the entropy term in the evidence lower bound for continuous variational inference. Its coordinate dependence must be combined consistently with that of the model density; changing units does not justify changing only one term of the objective.

Suppose one row of a finite discrete memoryless channel is a convex combination of other rows: $W(\cdot \mid x_0)={\sum }_j{\alpha }_{j}W(\cdot \mid x_j)$. Deleting input $x_0$ leaves the channel capacity unchanged. Given any input law with mass $t$ on $x_0$, move that mass to the other symbols in proportions ${\alpha }_j$. The output law stays the same. By the concavity of information entropy, $H(W(\cdot \mid x_0))\ge {\sum }_j{\alpha }_{j}H(W(\cdot \mid x_j))$, so this replacement cannot increase conditional entropy $H(Y\mid X)$ and cannot decrease mutual information. Taking maxima proves the claim; the smaller channel is also trivially a restriction of the original one. If the entropy inequality is strict, every positive use of the redundant row is strictly suboptimal compared with its redistribution.

For a finite alphabet source with independent and identically distributed symbols, a finite reconstruction alphabet, and a bounded nonnegative separable distortion, the rate-distortion function is the asymptotic limit for fixed-length lossy codes. In particular, for a distortion target in the interior of the feasible range and every $\epsilon >0$, sufficiently long block codes exist with rate at most $R(D)+\epsilon$ and expected distortion at most $D+\epsilon$. Every such $n$-symbol code obeys $R_n\ge R(D_n)$, where $D_n$ is its actual expected distortion.^[ref]

The outline is instructive. The data processing inequality bounds the mutual information in a reconstruction by the information entropy of the transmitted message, which is at most ${\log }_2{M}_n$. Independence and the convexity of the rate-distortion function then give the per-symbol lower bound. For achievability, choose many reconstruction blocks using a suitable probability distribution, so that the source's typical blocks are likely to have an acceptable nearby reconstruction. This is a covering argument; it does not require recovering the original block exactly.

For four uniformly distributed Bell states, measure one qubit along Bloch vector $\mathbf{n}$ and the other along $z$. The sign within each ${\Phi }^{\pm }$ or ${\Psi }^{\pm }$ pair is invisible and the displayed mutual information is at most one bit. A local Hadamard gate before a computational detector changes $n_z$ from one to zero and destroys the remaining information. An entangling Bell decoder falls outside this product-measurement restriction and recovers two bits.

A test channel is a proposed conditional distribution of a reconstruction given the source symbol. It describes a candidate joint probability distribution for evaluating mutual information and expected distortion. It need not be a physical transmission channel, or an implementable one-symbol encoder; the Shannon rate-distortion theorem relates this optimization to long block codes.

Total correlation measures how much dependence is lost when a joint probability distribution is replaced by the product of its marginal probability distributions. It compares the uncertainty of the separate coordinates with the uncertainty of the whole random vector. The distinction matters because an exact relation can involve three variables while leaving every pair independent.

For random variables $X_1,\dots ,X_n$ with finite alphabets, defineHere $H$ is information entropy, the entropy of the full random vector is joint entropy, and $D$ is relative entropy. Base-two logarithms give bits; natural logarithms give nats. The product law retains every marginal probability distribution but discards their dependence.^[ref] A positive joint probability guarantees positive marginal probabilities at that outcome, so the displayed relative entropy is finite in this finite-alphabet setting.

For two variables, total correlation is exactly mutual information. For three or more, it can detect a constraint that all pairwise mutual informations miss. Relative entropy nonnegativity gives $\mathrm{TC}\ge 0$, with equality exactly when the coordinates are jointly independent random variables.

As the simplest contrast, three independent fair bits have three bits of joint entropy and zero total correlation. If all three coordinates instead copy the same fair bit, their separate information entropies still sum to three bits, but their joint entropy is only one bit. Their total correlation is therefore two bits. The separate uncertainties count the same underlying choice three times.

For the finite alphabets considered here, the zero-rate distortion threshold is the smallest expected distortion obtainable by always outputting one fixed reconstruction. Such a reconstruction is independent of the source, so its mutual information is zero. A randomized output that is independent of the source averages the costs of fixed reconstructions and cannot improve their minimum. Thus $R(D)=0$ exactly when $D\ge D_{\mathrm{zero}}$. For a fair bit under Hamming distortion, $D_{\mathrm{zero}}=1/2$; for a Bernoulli random variable of parameter $p$, it is $\min \{p,1-p\}$. The threshold depends on the source probability distribution, rather than only the alphabet.