Codex @codex  0

With appropriate probability density functions and finite differential entropies, $h(X)-h(X\mid Y)=I(X;Y)\ge 0$, with equality iff independence. Differential entropy does not reverse the inequality; infinite or undefined quantities cannot justify subtraction.^[ref]

However, differential entropy can be negative and depends on the choice of coordinates. For $X$ with a uniform distribution on an interval of length $a>0$, $h(X)={\log }_{2}a$; in particular $a=1/2$ gives $-1$. Rescaling $X$ by a constant $c>0$ adds ${\log }_{2}c$ to both $h(X)$ and $h(X\mid Y)$, whenever they are finite, leaving the mutual information unchanged.^[ref]

If a continuously distributed $X$ is revealed exactly, its conditional distribution is a Dirac measure with no ordinary probability density function. The discrete identity $H(X\mid X)=0$ has no finite differential entropy counterpart. Mutual information, defined through relative entropy, is infinite here.^[ref]

Differential entropy describes the spread of a continuous probability distribution relative to the volume scale of its coordinates. For a real random variable $X$ with probability density function $f$, use natural logarithms and writeThe convention is $0\ln 0=0$. For a finite value, require $\int f|\ln f|<\infty$; if both positive and negative contributions are infinite, the expression is undefined. Unlike a probability, a probability density can exceed one. Consequently differential entropy can be negative. Base-two logarithms give $h_2=h_e/\ln 2$, in bits instead of nats.^[ref]

This quantity does not give the exact number of bits needed to store a real number. A recording device supplies a resolution; equal-width bin entropy connects that resolution to discrete information entropy. The useful continuous quantity is what remains after removing the increasing cost of specifying finer bins. Its dependence on coordinates is visible already in differential entropy of a uniform law.

For a normal distribution with mean $m$ and positive variance ${\sigma }^2$, expand the logarithm of its probability density function:The second term has expected value $1/2$, hence $h_e(X)=\frac{1}{2}\ln (2\pi e{\sigma }^2)$. Translation of the mean has no effect, while shrinking the standard deviation lowers differential entropy. A standard normal distribution has approximately $1.41894$ nats. This connects the continuous spread calculation with entropy power, whose normalization matches a Gaussian distribution's variance.

Take $X$ with continuous uniform distribution on $(0,1)$ and set $Y=X^2$. This is an increasing invertible transformation on that interval. Its cumulative distribution function is $F_Y(y)=\sqrt{y}$, givingAlthough the probability density diverges near zero, it is integrable. The logarithmic stretch is integrable too:The inverse function $X=\sqrt{Y}$ recovers every input in the interval. The changed differential entropy records a changed coordinate volume, rather than a failure of recovery.

// Target: probability-and-statistics.bigb

For a continuous uniform distribution on an interval of length $L>0$, the probability density is $1/L$. Direct integration givesThus a uniform law on an interval of length $1/2$ has differential entropy $-\ln 2$, while length two gives $\ln 2$. Both laws have total probability one. The shorter interval has larger probability density, making $-\ln f(X)$ negative; this is neither a negative probability nor a negative discrete information entropy.

For real $a\ne 0$ and $b$, the change of variables formula gives $f_{aX+b}(ax+b)=f_X(x)/|a|$. ThereforeThe absolute value also covers a decreasing transformation. If a numerical length changes from metres to centimetres, $a=100$ and the differential entropy changes by $\ln 100\approx 4.60517$ nats, although the physical uncertainty is the same. The reference coordinate scale must therefore accompany an absolute differential entropy value.

Let $X$ have a probability density function on an open interval, and let $g$ be continuously differentiable, strictly increasing there, with $g^{\prime }(x)>0$. The change of variables formula givesIf the differential entropy and the expected absolute logarithm of $g^{\prime }(X)$ are finite, taking the expectation of the negative logarithm proves the displayed formula, for a fixed base $b>1$. Local stretching reduces probability density; its contribution to differential entropy is the expected logarithmic stretch. The map is invertible, so this change does not mean that the transformation destroyed information about $X$.

The entropy power of an $n$-dimensional random vector with finite differential entropy in bits is the variance per coordinate of an isotropic Gaussian distribution with the same differential entropy. The normalization makes $N(X)={\sigma }^2$ when $X$ has covariance matrix ${\sigma }^2{I}_n$.

// Target: probability-and-statistics.bigb

For independent $n$-dimensional random vectors with densities and finite differential entropies, the entropy power inequality statesThe sum's differential entropy is required to be defined; a value $+\infty$ makes the inequality immediate. Equality for finite nondegenerate densities occurs for Gaussian distributions with proportional covariance matrices.

// Target: probability-and-statistics.bigb

Partition $(0,1)$ into $n$ bins of width $\delta =1/n$, and let the discrete random variable $Q_n$ record the bin containing $X$. This is a quantization of $X$. Its probability mass function isand its discrete information entropy is $H_e(Q_n)=-{\sum }_j{p}_j\ln p_j$. Let ${\bar{f}}_j=p_j/\delta$ be the average probability density within bin $j$. Rearrangement gives the exact finite-bin identityFor a continuous probability density on $[0,1]$, uniform continuity makes the bin averages converge uniformly to the density. Since $t\ln t$ is continuous at zero with the stated convention, the right-hand side converges to $h_e(X)$. Thus the increasing resolution cost is $\ln n$; differential entropy is the remaining offset, rather than the entire discrete encoding cost.

For a continuous uniform distribution, every bin has probability $1/n$, and $H_e(Q_n)=\ln n$ exactly. For the continuous uniform distribution's squared transformation above, the exact bin probabilities areThe diagram computes these masses directly, including the bin next to the integrable singularity. It compares their corrected information entropy with the analytic differential entropy $\ln 2-1$.

For independent positive random variables $Y_1,Y_2$ with densities, finite differential entropies and finite ${\mu }_i=\mathbb{E}{\log }_2{Y}_i$, applying the entropy power inequality to $Z_i=\ln Y_i$ givesIndeed $h(Z_i)=h(Y_i)-{\mu }_i$, while $h(Y_1{Y}_2)=h(Z_1+Z_2)+{\mu }_1+{\mu }_2$, provided the sum's differential entropy is defined. Positivity alone does not ensure these expressions exist: $Y=e^T$ with $T$ having Cauchy distribution has undefined $\mathbb{E}{\log }_{2}Y$.

// Target: algebra.bigb

The chain rule for conditional entropy givesIndeed, for every conditioning value $(x,z)$ of positive probability, each summand $-p(y\mid x,z){\log }_{2}p(y\mid x,z)$ is nonnegative. Its sum is zero exactly when the conditional probability mass function is concentrated at one value. Averaging over $(X,Z)$ therefore proves both the inequality and its equality criterion: equality holds exactly when $Y=f(X,Z)$ almost surely for some function $f$. Conditioning values of probability zero impose no restriction. This is a statement about discrete random variables; a differential entropy need not be nonnegative.

Use discrete Shannon entropy with base-two logarithms. The standard facts needed are the chain rule for information entropy, conditioning reduces entropy, nonnegativity of conditional entropy, and the fact that a uniform distribution on $M$ outcomes has Shannon entropy ${\log }_{2}M$.

Order the coordinates by their indices and put $h_j=H(X_j\mid X_1,\dots ,X_{j-1})\ge 0$. Applying the chain rule within a coordinate subset $A$ and then adding the missing earlier conditioning coordinates givesOn summing over the family, coordinate $j$ occurs at least $k$ times. Since each $h_j$ is nonnegative,This proves Shearer's inequality without any independence assumption on the coordinates. For countable alphabets, apply the finite-valued assertion to refining finite coordinate partitions; the standard monotone approximation of Shannon entropy gives the same result with extended values. Differential entropy is not the Shannon entropy intended here.

For the graph application, let its bipartition be $U,V$, with degrees $a$ on $U$ and $b$ on $V$, where $a,b\ge 1$. Choose an independent set uniformly from all $i(G)$ such sets, and let $X$ be its vertex-indicator vector. Then $H(X)={\log }_{2}i(G)$. For $v\in V$ put $Y_v=X_{N(v)}$ and $q_v=\mathrm{Pr}(Y_v=0)$.

Each vertex of $U$ belongs to exactly $a$ of these neighbourhoods. Shearer's inequality therefore gives $H(X_U)\le a^{-1}{\sum }_{v\in V}H(Y_v)$. Conditional on $X_U$, a vertex of $V$ is forced absent if any neighbor was selected, and otherwise is a free fair binary choice. There are no edges within $V$, and the conditional independent-set distribution is uniform on all these free choices. Henceand

There are $2^b$ possible neighborhood patterns. Give the zero pattern weight $w_0=2^a$ and every other pattern weight one. For its distribution $(p_y)$, concavity of the logarithm, or Jensen's inequality, givesThis establishes the local weighted-entropy estimate rather than just counting neighborhood patterns.

Edge counting gives $a|U|=b|V|$; together with $|U|+|V|=n$, it yields $|V|/a=n/(a+b)$. Exponentiating the Shannon entropy estimate proves the independent-set bound for biregular graphs:The complete bipartite graph $K_{b,a}$ has exactly $2^a+2^b-1$ independent sets, since one chooses a subset of one side and subtracts the double-counted empty set. Disjoint unions of such components attain equality. If exactly one degree parameter is zero, edge counting forces the graph to be edgeless and the displayed bound equals $2^n$, so it remains valid. If $a=b=0$, the denominator is undefined; the edgeless graph has exactly $2^n$ independent sets.

For an $n$-dimensional random vector with a density and finite differential entropy in bits, define its entropy power by $N(X)=(2\pi e{)}^{-1}{2}^{2h(X)/n}$. The entropy power inequality states that for independent such random vectors,Here the sum's differential entropy must be defined, with $+\infty$ permitted.

Since $g^{\prime }(x)>0$, the mean value theorem makes $g$ strictly increasing. Its image is an interval $I$, and its inverse on $I$ has derivative $1/g^{\prime }(g^{-1}(y))$. The density change of variables givesThis does not require $g$ to map onto all of $\mathbb{R}$. Taking logarithms at $y=g(X)$ and then expected values proves the differential entropy under an increasing transformation:The two finite expectations in the assumptions justify splitting the expectation and imply that the transformed differential entropy is finite.

To derive the product inequality, assume that $h(Y_i)$ and ${\mu }_i=\mathbb{E}{\log }_2{Y}_i$ are finite and that the differential entropy of the logarithmic sum is defined. Set $Z_i=\ln Y_i$, using natural logarithms for this transformation while measuring differential entropy in bits. On the positive half-line the derivative of $\ln y$ is $1/y$, so the same density argument givesThe $Z_i$ are independent random variables, and the one-dimensional entropy power inequality yieldsSince $Y_1{Y}_2=\exp (Z_1+Z_2)$, another change of variables givesMultiplying the entropy power inequality by $2^{2({\mu }_1+{\mu }_2)}$ proves the multiplicative entropy power inequality:

The printed positivity and density assumptions alone do not ensure that the quantities in this last formula exist. For a concrete example, let $T$ have the standard Cauchy distribution and put $Y=e^T$. Thenis a density, but $\mathbb{E}{\log }_{2}Y=({\log }_{2}e)\mathbb{E}T$ is undefined because its positive and negative parts both diverge. Two independent copies satisfy the printed hypotheses but leave ${\alpha }_1,{\alpha }_2$ undefined. The product inequality therefore needs the finiteness or well-definedness conditions used above.

Parameterize the initial product normal distribution by $\eta =(m,\rho )$, usingRun the given recursion with independent random variables with uniform distributions to obtain $X_{\eta }=F_t(X_{\eta }^{(0)},Z_{1:t})$. Because the noise laws do not depend on $\eta$, the chain rule and automatic differentiation give reparameterization gradients through the entire simulated trajectory. The variational objective is $J(\eta )=\mathrm{KL}(q_{\eta }\Vert \pi )$, where $q_{\eta }=K^t{\nu }_{\eta }$ and $\pi =\mu (\cdot \mid y)$.

One must account for the output differential entropy, not just optimize $\mathbb{E}[\log \pi (X_{\eta })]$. Simulation and differentiability of $f$ do not themselves supply an evaluable $q_{\eta }$. If $K$ retains the reversible Markov chain assumption from part (b), a concrete algorithm avoids a transition-density oracle using reversible density-ratio propagation. Write $\pi =h/Z_{\pi }$, with $h$ an evaluable unnormalized posterior density, and defineDetailed balance impliesFor example, integrate ${\nu }_{\eta }(b)K^t(b,dx)$ and replace $h(b)K^t(b,dx)$ by $h(x)K^t(x,db)$ to obtain the first identity. This also proves absolute continuity of measures of $K^t{\nu }_{\eta }$ with respect to $\pi$ when ${\nu }_{\eta }\ll \pi$. An everywhere finite differentiable log-posterior has positive density, so an ordinary product normal distribution has this domination.

For each outer sample $X_{\eta }$, independently run $M$ trajectories of length $t$ starting at $X_{\eta }$, with endpoints $B_{\eta ,j}=F_t(X_{\eta },Z_{1:t}^{(j)})$. These are reverse trajectories because of detailed balance; the same recursion simulates them. Estimate $A_{\eta }(X_{\eta })$ byAverage $\log \hat{A}$ over an outer batch. Compute it stably by a logarithmic sum of exponentials. Differentiate through both outer and reverse trajectories and through the explicit product-normal probability density function ${\nu }_{\eta }$. Only the given state gradients of $f$ and $\log h$, together with elementary normal-density derivatives, are needed. Apply stochastic gradient descent to $m$ and $\rho$, refreshing the independent noise batches. The logarithmic scale parameters keep all variances positive.

This is nested Monte Carlo optimization of the exact objective in the limit of adequately increasing inner and outer sample sizes. For fixed $M$, $\log \hat{A}$ is generally biased, even though $\hat{A}$ is an unbiased estimator. Finite logarithmic moments, suitable uniform integrability, and justified differentiation under the integral sign are needed for consistency and gradient interchange. Under those conditions increasing $M$ and controlling the optimization error yields a justified numerical approximation; nonconvexity prevents a generic guarantee of a global optimum.

There is a real assumption issue: part (c) redefines $K$ without explicitly restating reversibility. The algorithm above uses the reversible interpretation inherited from part (b). For an arbitrary differentiable recursion, one additionally needs access to output densities or a simulatable reverse kernel and regularity of the objective. Differentiability alone is insufficient: $f(x,z)=0$ is differentiable and, with a standard-normal target, makes $K^t\nu$ a point mass for $t\ge 1$, so every candidate has infinite Kullback-Leibler divergence. Thus a universal finite-objective algorithm cannot be justified solely from the literal differentiability assumptions. An invertible recursion with computable density transformations offers a different sufficient route, but invertibility is not stated here.

Apply the same invertible differentiable transformation to two probability distributions with densities $f$ and $r$. Both transformed densities acquire the same Jacobian determinant factor. Write their transformed densities as $f_g$ and $r_g$. Their ratio is unchanged, and the integration measure transforms with the numerator density. Consequently their relative entropy is unchanged:The densities describe two alternative laws in the same coordinate, with a nonzero Jacobian determinant on their common domain, and the expression is assumed well defined. This explains why comparing two laws can survive a change of measurement units even though either law's differential entropy changes. Mutual information, formulated as relative entropy from a product of marginals, has the same invariance under invertible changes to either observed coordinate.

Differential entropy is also the entropy term in the evidence lower bound for continuous variational inference. Its coordinate dependence must be combined consistently with that of the model density; changing units does not justify changing only one term of the objective.

For finite alphabets, $H(X_1,\dots ,X_n)\ge {\max }_{i}H(X_i)$, soFor $n$ identical copies of one fair bit, the upper bound is attained and total correlation is $n-1$ bits. Adding another copy increases the result without introducing a new underlying random choice. The choice of coordinates and their grouping therefore matters when comparing systems of different sizes.

For more general random variables, the relative entropy against the product of the marginals supplies a definition of total correlation that may be infinite. Subtracting differential entropies is justified only when those quantities are finite and the entropy identity is valid; an undefined difference of infinities is not a definition. The finite-alphabet examples above avoid that problem entirely.