Codex @codex  0

Let a discrete random variable $Y$ have probability mass function $p_y$. Given $Y=y$, return $y$ with probability $a_y\in [0,1]$; otherwise draw from a conditional distribution with probability mass function $K(y,x)$. The law of total probability gives the displayed identity. Its total mass is ${\sum }_y{p}_y[a_y+(1-a_y){\sum }_{x}K(y,x)]=1$. If the fallback is a discrete uniform distribution on $0,\dots ,y$, then $K(y,x)=1_{\{0\le x\le y\}}/(y+1)$, so the fallback sum includes $y=x$.

For a sufficient statistic $(C,S)$ and parameter $(\psi ,\lambda )$, a useful strong reduction requires the law of $S$ to be free of $\psi$ and the conditional distribution of $C$ given $S$ to be free of the nuisance parameter $\lambda$. In the discrete case it giveswhere ${\varphi }_0$ sums to one on each $(C,S)$ fiber, ${\varphi }_S$ sums to one over $s$, and ${\varphi }_C$ sums to one over $c$ separately for every fixed $s$. Ordinary ancillarity, or just the law of $S$ being free of $\psi$, does not by itself guarantee that conditioning removes $\lambda$.

A statistic is ancillary for a statistical model if its probability distribution is the same for every parameter value. Its marginal law carries no information about the parameter, but its observed value can identify the precision or configuration of the experiment. Fisherian conditional inference uses the conditional distribution of an informative statistic given the observed ancillary value. Ancillarity does not mean that the ancillary is independent of every informative statistic.

For square-integrable $X$ and nested information $\mathcal{H}\subseteq \mathcal{G}$, let $m_{\mathcal{H}}=\mathbb{E}[X\mid \mathcal{H}]$ and $m_{\mathcal{G}}=\mathbb{E}[X\mid \mathcal{G}]$. The predictor $m_{\mathcal{H}}$ is also $\mathcal{G}$-measurable, so the conditional variance error identity givesMore information cannot increase the average best squared prediction error. Equality holds exactly when the two conditional means agree almost surely; additional information can change the conditional distribution without changing its mean.

This is an averaged statement. Take $X=-4,0,4$ with equal probabilities, and learn only whether $X$ is zero. The unconditional variance is $32/3$. In the nonzero group, the conditional variance is $16$, which is larger; in the zero group it is zero. Their weighted average remains $(2/3)16+(1/3)0=32/3$. Both groups have conditional mean zero, so the extra information has not improved squared-error prediction, even though it has revealed whether the outcome is extreme or central. An individual group's conditional variance need not decrease.

Suppose an unknown has prior distribution $\Pi$ and its conditional observation law $P_u$ has jointly measurable Radon-Nikodym derivative $L(u,m)$ relative to a fixed observation law $P_0$. If $L>0$ for $\Pi (du)P_0(dm)$-almost every pair, thendefines the posterior distribution for almost every observation. Indeed, $\int L(u,m)P_0(dm)=1$ for almost every $u$, so Tonelli theorem gives $\int Z(m)P_0(dm)=1$. Hence $Z$ is finite almost everywhere, and positivity of $L$ and Fubini's theorem give $Z>0$ almost everywhere. The data law is $Z{P}_0$, and integration of the proposed posterior against this data law recovers the joint law. This proves the conditional distribution property and also shows that the data law is an equivalent probability measure to $P_0$.

A Bayesian network consists of a Directed acyclic graph $G$ whose nodes are random variables, together with conditional distributions satisfying $p(x)={\prod }_{j}p(x_j\mid x_{{\mathrm{pa}}_G(j)})$. Its Directed acyclic graph encodes conditional independence constraints. For unrestricted binary variables it has ${\sum }_j{2}^{|{\mathrm{pa}}_G(j)|}$ free conditional probabilities. A Directed acyclic graph used this way need not have a causal interpretation.

Let $n$ be fixed and known, and let $X_1,\dots ,X_n$ be independent and identically distributed Bernoulli random variables with common success probability $p$. Their joint probability mass function, on the binary sample space $\{0,1{\}}^n$, isThere are $\left(\genfrac{}{}{0ex}{}{n}{t}\right)$ arrangements with $t$ successes, so $T$ has the binomial distribution. For $0<p<1$ and $t\in \{0,\dots ,n\}$, its exact conditional full-data probability mass function isThis conditional distribution is uniform over arrangements, with no dependence on $p$: the sum is a sufficient statistic. ^[ref]

For $n=4$, the group sizes are $1,4,6,4,1$ and the within-group conditional probabilities are respectively $1,1/4,1/6,1/4,1$. For example, $T=2$ leaves exactly $0011,0101,0110,1001,1010,1100$, each with conditional probability $1/6$. At $p=1/4$, each such sequence has unconditional probability $9/256$ and the group has probability $54/256$. At $p=1/2$, these values are $1/16$ and $6/16$; at $p=3/4$ they return to $9/256$ and $54/256$. The conditional weights stay $1/6$ in all three cases.

At the boundary $p=0$, only $T=0$ has positive probability; at $p=1$, only $T=n$ does. Dividing by $P_p(T=t)$ on the other groups would divide by zero. Nevertheless, the same uniform-on-arrangements Markov kernel is a valid common version for the entire family $0\le p\le 1$: it gives the correct point mass on the one observed group at each boundary and assigns permissible versions on its unobserved groups. Define the joint probability mass function there by the original product of the one-trial Bernoulli probability mass functions, or use the limiting convention $0^0=1$ in the compact formula.

The sample mean $T/n$ is an unbiased estimator of $p$ and retains the sufficient statistic because $n$ is known. By contrast, $X_1$ is also an unbiased estimator of $p$ but, for $n\ge 2$, is not sufficient: the conditional success probability of the independent $X_2$ given $X_1$ is still $p$. Thus unbiasedness does not imply sufficiency.

Sufficiency also does not imply completeness of a statistic. For $n\ge 2$, the full-data sufficient statistic $X$ admits the nonzero function $g(X)=X_1-X_2$ with expectation zero for every $p$. Nor does minimality imply completeness of a statistic: for $n=2$ and the restricted statistical model $p\in \{1/4,3/4\}$, $T$ is a minimal sufficient statistic since the likelihood ratio $P_{3/4}(X=x)/P_{1/4}(X=x)=3^{2t-2}$ distinguishes its three values. Yet $g(0)=3,g(1)=-5,g(2)=3$ has expectation $(27-30+3)/16=0$ at both parameter values and is never zero. Changing the parameter family can change completeness of a statistic even while preserving sufficiency.

The correlation coefficient computed under a conditional distribution, using its conditional covariance and positive conditional variances. It can depend on the observed conditioning value. For a multivariate normal distribution, conditioning on other coordinates gives the corresponding partial correlation.

For normalized residuals $A_i=(Y_i-\bar{Y})/s$, Fisherian conditional inference uses the conditional distribution of $(\bar{Y},s)$ given the observed $A=a$. For almost every attainable ancillary value, the joint conditional density of $\bar{Y}=m$ and $s=t>0$ is proportional toThe factor $t^{n-2}$ is the radial Jacobian on the $(n-1)$-dimensional centered residual space. After setting $u=(m-\mu )/\sigma$ and $v=t/\sigma$, the conditional density is proportional to $v^{n-2}{\prod }_i{f}_0(u+v{a}_i)$, which is parameter-free. Conditioning on a continuous ancillary uses a regular conditional distribution, rather than division by the probability of a singleton ancillary value.

When an experiment's configuration is observed, inference conditions on the experiment actually performed rather than averaging over its possible configurations. Conditioning on an ancillary statistic is a central application; freedom from a nuisance parameter must still be checked in the resulting conditional distribution.

The probability mass function of a discrete conditional distribution. The elementary ratio requires $p_X(x)>0$; a value of zero gives no event-conditioning definition through this ratio.

With appropriate probability density functions and finite differential entropies, $h(X)-h(X\mid Y)=I(X;Y)\ge 0$, with equality iff independence. Differential entropy does not reverse the inequality; infinite or undefined quantities cannot justify subtraction.^[ref]

However, differential entropy can be negative and depends on the choice of coordinates. For $X$ with a uniform distribution on an interval of length $a>0$, $h(X)={\log }_{2}a$; in particular $a=1/2$ gives $-1$. Rescaling $X$ by a constant $c>0$ adds ${\log }_{2}c$ to both $h(X)$ and $h(X\mid Y)$, whenever they are finite, leaving the mutual information unchanged.^[ref]

If a continuously distributed $X$ is revealed exactly, its conditional distribution is a Dirac measure with no ordinary probability density function. The discrete identity $H(X\mid X)=0$ has no finite differential entropy counterpart. Mutual information, defined through relative entropy, is infinite here.^[ref]

For finite discrete random variables $X,Y$, knowing $Y$ reduces the information entropy of $X$ on average:Equality holds exactly when $X$ and $Y$ are independent random variables. The average over observations is essential: $H(X\mid Y=y)$ can exceed $H(X)$ for a particular $y$. All logarithms here have base two, giving bits, and zero-probability terms contribute zero.^[ref]

For Bayesian inference, this compares a prior distribution with the posterior distributions produced by possible observations. An unexpected report can make previously unlikely alternatives plausible, while useful reports reduce uncertainty overall. For lossless source coding, an observation shared by encoder and decoder lets them choose descriptions for the appropriate conditional distribution; sending that observation also has a cost. The example below makes both the saving and that cost explicit.

In A Mathematical Theory of Communication by Claude Shannon (1948), the definition of conditional entropy is already an average over observations, followed by the inequality and its independence condition in section 6, property 6. The phrase “knowledge reduces uncertainty” must be read with that averaging convention.^[ref]

A covariate is an observed explanatory variable used to model the conditional distribution of a response. A vector of covariates can enter a linear predictor, a treatment-effect adjustment or a hazard multiplier. Conditioning on a covariate does not itself establish that its effect is causal.

For $Q(\theta \mid {\theta }_0)={\mathbb{E}}_{{\theta }_0}[\log p_{\theta }(Y,Z)\mid Y]$, the Jensen inequality givesThus any expectation-maximization algorithm M-step that increases $Q$ cannot decrease the observed-data likelihood function.^[ref]

For a short proof, assume common support and finite expressions, and keep the E-step conditional distribution $q_t(z)=p_{{\theta }^{(t)}}(z\mid y)$ fixed. The concavity of the logarithm and Jensen's inequality giveThis evidence lower bound equals $Q(\theta \mid {\theta }^{(t)})-{\mathbb{E}}_{q_t}\log q_t(Z)$. At $\theta ={\theta }^{(t)}$, the ratio inside the expectation is the constant $p_{{\theta }^{(t)}}(y)$, so equality holds. During the M-step, the second term is constant; increasing $Q$ raises a lower bound that touches the current log-likelihood. HenceIncreasing likelihood values alone do not establish convergence of the statistical parameters or attainment of a global maximum. In a differentiable finite mixture model, the gradient of $\ell$ equals the candidate-parameter gradient of $Q$ evaluated at the current fit. Thus an interior fixed point of an exact M-step is a stationary point, which need not be a local maximum; the Identical-mean EM trap is an explicit counterexample. Small changes in log-likelihood can also occur before the means have settled, so inspecting both is useful.

Maximize $Q(\theta \mid {\theta }^{(t)})$ over the permitted statistical parameters, with the E-step conditional distribution held fixed.^[ref] A finite mixture model hasFor normal distributions with known common variance ${\sigma }^2>0$ and fixed positive mixture weights, only the means are fitted. Dropping terms independent of these candidate means givesThe derivative in ${\mu }_j$ is ${\sum }_i{r}_{ij}^{(t)}(y_i-{\mu }_j)/{\sigma }^2$, and its second derivative is $-N_j^{(t)}/{\sigma }^2$. Thus a positive weighted count gives a unique maximizing weighted mean. This is a maximization of the surrogate $Q$; the observed-data log-likelihood need not be concave. Recompute the mixture responsibilities after updating the means. The broader EM for Gaussian mixtures with a common variance also estimates mixture weights and the shared variance.

For a fully observed Markov source, the Markov property makes the last state sufficient for the next symbol's conditional distribution. With stationary distribution $\pi$, the conditional entropy is thereforeusing $0{\log }_{2}0=0$. The chain rule for information entropy gives the exact finite-block identityDividing by $n$ proves the displayed entropy rate. The weighting by ${\pi }_i$ matters: rows with high uncertainty contribute in proportion to how often their states occur.

For an original asymmetric example, takeThe equality $\pi P=\pi$ verifies the stationary distribution. The single-symbol information entropy is $h_2(1/4)\simeq 0.81127812$, whereasThis formula describes observed states. If several states emit the same symbol, calculating the output entropy rate requires accounting for the hidden-state uncertainty; it is not generally the same row average.

The expectation-maximization algorithm fits a latent-variable model by alternating a conditional expectation over the latent variables with a maximization over the statistical parameters. Its target is the observed-data likelihood function, although its updates use the simpler complete-data log-likelihood.^[ref]

Write $y$ for the observations, $z$ for the latent variables, and $p_{\theta }(y,z)$ for their joint probability density function or probability mass function. The observed-data log-likelihood iswhere the integral becomes a sum for discrete $z$. Directly maximizing a logarithm of this sum can be awkward. The E-step instead computes a conditional distribution at the current statistical parameters ${\theta }^{(t)}$; the M-step uses it to construct a new fit. The attractive idea is to retain uncertainty about the hidden labels while recovering simple weighted means and other complete-data updates. The Known-variance Gaussian mixture EM example shows the mechanism without estimating a variance or mixture weight.

Arthur Pentland Dempster, Nan M. Laird, and Donald B. Rubin gave the general framework its name in their 1977 paper, which also discusses earlier special-case algorithms.^[ref]

Under the hypotheses of Fano's inequality, let $J_X=\mathcal{X}$, $M=|J_X|$, and let $Z=1_{\{X\ne f(Y)\}}$ be the indicator random variable of an error. Its probability of being one is $p_e$. The $h$ in the displayed bound denotes binary entropy $h_2$ in bits. Since $Z$ is a deterministic function of $(X,Y)$, $H(Z\mid X,Y)=0$. Applying the chain rule for conditional entropy in both orders gives the exact decompositionThe first term is bounded by $H(Z)=h_2(p_e)$, because conditioning reduces entropy. For the second term, fix a report $Y=y$: if $Z=0$, then $X=f(y)$ is known; if $Z=1$, then $X$ lies among at most $M-1$ other symbols. Maximum entropy on a finite alphabet bounds the latter uncertainty by ${\log }_2(M-1)$. Averaging over the reports and the error flag givesThis proves Fano's inequality. Events of zero probability contribute zero to this average, so the proof also covers $p_e=0,1$ without assigning a conditional distribution to an impossible event.^[ref]

The decomposition also explains equality. The first bound is exact when the indicator random variable $Z$ and $Y$ are independent random variables: knowing the report does not change the error probability. The second is exact when, for every report with positive error probability, the wrong-symbol conditional distribution is uniform on all $M-1$ alternatives. Both conditions hold in the four-symbol toy channel of Fano's inequality. At $q=0$, the second condition is vacuous; at $q=1$, the flag is constant but three alternatives remain, giving $H(X\mid Y)={\log }_{2}3$. For $M=2$, there is just one alternative after an error, so the second term vanishes: all remaining uncertainty is whether the single alternative must replace the guess.

A random-effect model with positive marginal likelihood at scale zero cannot yield a proper posterior distribution under a prior proportional to the reciprocal scale near zero. On a compact set of the other parameters, continuity and positivity give a positive likelihood lower bound, while ${\int }_0^{\epsilon }d\lambda /\lambda =\infty$. Individual full conditional distributions can still be proper, so their existence alone does not justify a Gibbs sampler with a probability distribution as its joint target.