Codex @codex  0

Aaditya Ramdas develops statistical inference methods that remain valid when data are examined sequentially. His research also includes multiple testing and machine learning. A central practical question is how to report evidence while allowing data collection to continue or stop in response to the observations.^[ref]

He coauthored distillation for the conditional randomization test, combining conditional independence testing with flexible test statistics and reduced computational cost. The calibration relies on the assumed feature conditional distribution.^[ref]

For finite discrete random variables $X,Y,Z$, retaining $Y$ and also observing $Z$ givesEquality means conditional independence of $X$ and $Z$ given $Y$. Both sides average over reports; the additional observation is retained alongside $Y$.^[ref]

A short proof of the original inequality uses Kullback-Leibler divergence:Gibbs inequality makes each term nonnegative, vanishing exactly when $p_{X\mid Y=y}=p_X$: independence for every possible report. Applying this within each $Y=y$ and averaging proves the additional-conditioning inequality.^[ref]

A Bayesian network consists of a directed acyclic graph $G$ whose nodes are random variables, together with conditional distributions satisfying $p(x)={\prod }_{j}p(x_j\mid x_{{\mathrm{pa}}_G(j)})$. Its directed acyclic graph encodes conditional independence constraints. For unrestricted binary variables it has ${\sum }_j{2}^{|{\mathrm{pa}}_G(j)|}$ free conditional probabilities. A directed acyclic graph used this way need not have a causal interpretation.

For a finite measurable partition $(B_j)$ with positive probabilities, and an event $A$ of positive probability,Its beauty is that a model predicting observations from a hidden state can also describe uncertainty about the state after an observation. Each numerator is a joint probability; the denominator, supplied by the law of total probability, makes the answers sum to one. This is a direct consequence of conditional probability, rather than an extra axiom.

Select the coin from law of total probability once, then toss that same coin repeatedly, with conditionally independent tosses given its identity. Let $B$ denote the $3/4$-head coin, initially of prior probability $1/5$. After one head $H$, its posterior probability isThe likelihood of a head is larger under $B$, but $B$ remains less likely than the other coin because its prior probability was smaller. A large likelihood for an observation does not itself imply a large posterior probability for the hypothesis.

After an ordered sequence containing $h$ heads and $t$ tails, its likelihood ratio between the two coins is $(3/2{)}^h(1/2{)}^t$. Bayes' theorem therefore gives posterior oddsEach head multiplies the odds by $3/2$ and each tail by $1/2$. Three heads give posterior probability $27/59$, still below one half; four give $81/145$, above one half. The finite-model update and the distinction between prior probability, likelihood and posterior probability are set out in MIT 18.05, Reading 11: Bayesian Updating with Discrete Priors.

The hidden selection also creates dependence. Each toss has marginal head probability $11/20$, but two heads have joint probability $(4/5)(1/2{)}^2+(1/5)(3/4{)}^2=5/16$, greater than $(11/20{)}^2=121/400$. Their indicator covariance is $1/100$. Seeing a head makes the head-favoring coin more likely, so it also raises the chance of a future head. This is covariance induced by a shared latent variable, despite conditional independence of the tosses given the coin.

Historically, Thomas Bayes's essay on an unknown success probability appeared in 1763 after his death, communicated by Richard Price. Its opening problem asks how observed successes and failures constrain that unknown probability; Price's introductory letter explains the connection to reasoning from experience. That is a more substantial inference problem than simply rearranging today's elementary identity. ^[ref]

A causal directed acyclic graph represents variables by vertices and direct causal relations by arrows. Its graphical separation rules encode conditional independences implied by the causal model.

Causal discovery seeks information about causal structure from observations or interventions, rather than assuming the complete structure in advance. In a causal directed acyclic graph, observed conditional independences can restrict possible arrows, but generally do not identify a unique graph without further assumptions.

For three variables, the chains $X\to Y\to Z$ and $X\leftarrow Y\leftarrow Z$, and the fork $X\leftarrow Y\to Z$, all imply $X$ and $Z$ are independent conditional on $Y$ in the corresponding faithful models. These independences alone cannot distinguish those structures. By contrast, the collider $X\to Y\leftarrow Z$ generically makes $X$ and $Z$ marginally independent but dependent after conditioning on $Y$.

This example explains both the appeal and the limitation of searching for graphical structure. Conditional independence tests offer clues, while hidden common causes, selection and unfaithful parameter cancellations can undermine the inference. An experiment that changes $X$ can distinguish directions that observational data alone leave ambiguous.^[ref]

Let $E_1,E_2,E_3$ be independent random variables with the standard normal distribution and set $Z_1=E_1$, $Z_2=Z_1+E_2$, $Z_3=Z_2-Z_1+E_3$. The complete acyclic graph with edges $1\to 2$, $1\to 3$, $2\to 3$ is causally minimal: deleting each edge imposes a false marginal or conditional independence. Yet $Z_1\perp Z_3$ because $Z_3=E_2+E_3$, so faithfulness of a directed acyclic graph fails. Nonzero structural coefficients alone do not ensure faithfulness.

A probability distribution is causally minimal for a directed acyclic graph when it satisfies the global Markov property for a directed acyclic graph for that graph and for no proper subgraph obtained by deleting edges on the same vertex set. It excludes redundant edges without excluding every extra conditional independence.

A coeval binary has components formed at the same time. A population of such binary stars can still have a distribution of system ages. If component masses have independence and the state at age $t$ is determined separately for each component, the states have conditional independence given $t$. If $g(t)$ and $w(t)$ are the conditional probabilities of two states, the unconditional unordered pair fractions are $\mathbb{E}[g(t{)}^2]$ and $2\mathbb{E}[g(t)w(t)]$, not generally products of marginal state fractions.

Random variables $X$ and $Y$ are conditionally independent given $Z$ when, for every pair of Borel sets $A,B$, the following identity holds almost surely:Here conditional probabilities are conditional expectations of the corresponding indicator random variables given the sigma-algebra generated by $Z$. This defines the condition even when $\mathbb{P}(Z=z)=0$. In terms of regular conditional distributions, the conditional joint law is the product of the conditional marginal distributions for almost every conditioning value.

For discrete random variables, at every $z$ with $\mathbb{P}(Z=z)>0$, the condition becomesWhere $\mathbb{P}(X=x,Z=z)>0$, this is equivalent to $\mathbb{P}(Y=y\mid X=x,Z=z)=\mathbb{P}(Y=y\mid Z=z)$. Once $Z$ is known, observing $X$ provides no further information about $Y$.^[ref]

For a worked example, suppose a latent variable $T$ chooses between two environments with $\mathbb{P}(T=0)=\mathbb{P}(T=1)=1/2$. Given $T=t$, take $X,Y$ to be independent and identically distributed with a Bernoulli distribution of parameter $p_t$, where $p_0=0.1$ and $p_1=0.9$. Think of two separate trials run under the same quiet or active environment. Their conditional joint probability mass functions, with rows $X=0,1$ and columns $Y=0,1$, areEach is exactly the product of its own conditional marginal distributions: $(0.9,0.1)$ for $T=0$, and $(0.1,0.9)$ for $T=1$. Thus $X,Y$ are conditionally independent given $T$.

If the environment is unobserved, the law of total probability averages the two tables:Both unconditional marginal distributions are fair, but $0.41\ne 0.5\times 0.5$. The law of total expectation gives

Observing $X=1$ makes the active environment more plausible, so it raises $\mathbb{P}(Y=1\mid X=1)$ to $0.41/0.5=0.82$. Knowing the actual environment removes this source of statistical dependence. The covariance induced by a shared latent variable lemma explains the calculation without listing all the outcomes.

This is why conditional independence is useful in statistical modelling: a model can multiply simple conditional laws and then average over a shared latent variable. The resulting observations can remain statistically dependent. Conditional independence graphs organize such information relations; their interpretation as causes requires additional assumptions from causal inference. Conditioning describes the information available to an observer. It does not, by itself, describe what would happen if an experimenter changed a variable.

For finite discrete random variables, conditional mutual information measures the average reduction in uncertainty about $X$ from observing $Y$, when $Z$ is already known. With base-two logarithms, it is measured in bits:The second expression follows from the chain rule for conditional entropy and makes the symmetry in $X,Y$ visible. Equivalently, compute the mutual information separately in each conditional probability distribution and then average:Here $D$ is the Kullback-Leibler divergence, also computed with base-two logarithms; zero-probability terms contribute zero. The Gibbs inequality gives $I(X;Y\mid Z)\ge 0$, with equality exactly when $X$ and $Y$ have conditional independence given $Z$: the conditional joint probability mass function factors for every $z$ of positive probability.^[ref]

For a fully computed example, take independent random variables $X,Y$ that are fair bits, and let $Z=X\oplus Y$, where $\oplus$ is exclusive or. The complete joint probability distribution has only these four positive entries:

Every pair $(x,y)$ has probability $1/4=p_X(x)p_Y(y)$, so $I(X;Y)=4(1/4){\log }_{2}1=0$. But $p_Z(0)=p_Z(1)=1/2$. Given $Z=0$, the pairs $(0,0),(1,1)$ each have conditional probability $1/2$, while both conditional marginals are fair. ThusGiven $Z=1$, the pairs $(0,1),(1,0)$ give the same calculation. ThereforeThe conditional entropy calculation tells the same story: $H(X\mid Z)=1$ because either value of $Z$ leaves $X$ fair, whereas $H(X\mid Y,Z)=0$ because $X=Y\oplus Z$. Learning the exclusive or does not reveal either bit alone; it reveals whether they agree, so learning one then determines the other.

This example distinguishes conditional independence from ordinary independence. It also disproves the claim that conditioning always reduces mutual information, despite conditioning reduces entropy. There is no general ordering between $I(X;Y\mid Z)$ and $I(X;Y)$: the example above increases it from zero to one, while taking $Y=X$ and $Z=X$ for a fair bit decreases it from one to zero. The chain rule for mutual information explains why both are possible.^[ref]

Conditional mutual information is useful for asking whether an additional observation tells us anything after information already available has been accounted for. Its zero case expresses conditional independence exactly; in a Markov chain, this is the step that turns the chain rule for mutual information into the data processing inequality.

A conditional randomization test tests $Y\perp X\mid Z$ by resampling the tested variable from its conditional law given the remaining covariates. For independent sample rows, hold the observed $Y,Z$ fixed and draw each replacement vector $X^{(b)}$ from the true product conditional law of $X\mid Z$. Under the null hypothesis, the original $X$ and these replacements are conditionally exchangeable. Applying the same prespecified test statistic and fitting rule to every dataset therefore gives the valid rank-based Monte Carlo testThe add-one convention and conservative treatment of ties give finite-sample type I error control. Other resampling constructions are possible when they preserve the same conditional exchangeability.^[ref]

Replacing the true conditional law by an estimate does not automatically preserve that guarantee. For a counterexample without $Z$, let the true $X$ be a fair Bernoulli random variable and let $Y$ be constant, so independence holds. If the resampling model incorrectly uses $P(X=1)=0.99$ and the test statistic is $T=1-X$, the model assigns upper-tail probability $0.01$ whenever the observed $X=0$. That event actually occurs with probability $0.5$, so the limiting simulation test rejects far more often than level $0.05$ permits.

Estimated conditional models require a separate calibration or approximation argument. More resamples reduce Monte Carlo error but do not correct a wrong conditional law. The method tests conditional independence; interpreting it as a causal effect test requires the appropriate additional causal inference assumptions.

Let $X,Y$ be independent random variables, each with a Bernoulli distribution of parameter $1/2$, and put $S=X+Y$. Unconditionally, the four ordered pairs $(0,0),(0,1),(1,0),(1,1)$ each have probability $1/4$. On the event $S=1$, only $(0,1)$ and $(1,0)$ remain, each with conditional probability $1/2$. ThusKnowing that exactly one trial succeeded makes observing one outcome informative about the other. This violates conditional independence given $S$, even though the original random variables are independent. Given $S=0$ or $S=2$, both variables are constant random variables and are conditionally independent there; that does not repair the failure at $S=1$, an event of positive probability.

The conditional covariance here is $\mathbb{E}[XY\mid S=1]-\mathbb{E}[X\mid S=1]\mathbb{E}[Y\mid S=1]=0-1/4=-1/4$. Both conditional variances are $1/4$, so the conditional correlation coefficient is $-1$: within the selected group, $Y=1-X$. These conclusions follow from restricting a finite sample space, without positing any physical interaction between the trials.

This example and the shared-environment example establish both logical directions: independent random variables need not be conditionally independent, and conditionally independent variables need not be independent. When $S$ is represented as the joint result of two inputs in a causal directed acyclic graph, conditioning on it has the collider structure associated with collider bias. A probability calculation by itself establishes a dependence relation, not a physical causal mechanism or an intervention.

Let $A,B$ be indicator random variables with conditional independence given a latent variable $T$, and suppose $\mathbb{E}[A\mid T]=\mathbb{E}[B\mid T]=g(T)$. ThenFor a short proof, conditional independence gives $\mathbb{E}[AB\mid T]=g(T{)}^2$. The law of total expectation gives $\mathbb{E}[AB]=\mathbb{E}[g(T{)}^2]$ and $\mathbb{E}[A]=\mathbb{E}[B]=\mathbb{E}[g(T)]$. Subtract their product to obtain the claimed variance. Since $0\le g(T)\le 1$, all the required expected values exist.

In the two-environment example of conditional independence, $g(T)$ is $0.1$ or $0.9$ with equal probabilities. Hence $\mathbb{E}[g(T)]=0.5$ and $\mathrm{Var}(g(T))=((0.1-0.5{)}^2+(0.9-0.5{)}^2)/2=0.16$. The two trials have zero conditional covariance in either environment, yet their unconditional covariance is positive because their success probabilities rise and fall together.

More generally, square-integrable conditionally independent random variables satisfyby the law of total covariance. Equal conditional means reduce this to the displayed variance; unequal conditional means need not give a nonnegative covariance. The equal-mean result explains correlated component evolutionary states in a coeval binary population without correlation between initial masses: variation in a shared environment can supply the entire covariance.

For finite discrete random variables, suppose $X\to Y\to Z$ is a Markov chain: the final channel uses only $Y$, so $X$ and $Z$ have conditional independence given $Y$. In terms of the joint probability mass function,and hence $I(X;Z\mid Y)=0$. Then the two data processing inequalities areAll mutual informations here use base-two logarithms. Thus deterministic or randomized processing cannot increase the information shared with an unprocessed variable.^[ref]

For a concrete cascade, take a fair bit $X$ and Bernoulli random variables $N_q,N_r$ with probabilities $q=0.1$, $r=0.2$ of being one. Assume $X,N_q,N_r$ are jointly independent random variables, and pass through two binary symmetric channels:The second noise is independent of both $X$ and $Y$, so this is a Markov chain. Both $Y$ and $Z$ are fair bits. The net exclusive or error is one exactly when one noise flips the bit and the other does not:Using the binary entropy $h_2(t)=-t{\log }_{2}t-(1-t){\log }_2(1-t)$, the three mutual informations are fully determined:All numbers are in bits. The two losses are $I(X;Y\mid Z)\approx 0.3577507789$ and $I(Y;Z\mid X)\approx 0.1048182776$. Thus the cascade retains less information about the original source than either individual channel shares between its own input and output.

With $q=0.1$ fixed and $0\le r\le 1/2$, the effective error is $s=0.1+0.8r$. At $r=0$, $Z=Y$ and $I(X;Z)=I(X;Y)$; at $r=1/2$, $Z$ is independent of the previous variables and $I(X;Z)=I(Y;Z)=0$. The diagram plots these endpoints and the computed $r=0.2$ example.

The data processing inequality is useful for testing what a proposed decoder or data summary could possibly reveal: if its only input is $Y$, even randomized processing cannot provide more mutual information about $X$ than $Y$ already contains. The Markov chain hypothesis matters. In the exclusive or example from conditional mutual information, allowing a processor to use both $Y$ and $Z$ reveals $X$ exactly; that processor has additional input, so the conclusion for processing $Y$ alone does not apply.

The chain rule for mutual information proves both inequalities and identifies their exact gaps. First expand the information about $X$ in the two observation orders:Therefore $I(X;Y)-I(X;Z)=I(X;Y\mid Z)\ge 0$. Next use $Z$ as the variable being learned:Symmetry of mutual information gives $I(Y;Z)-I(X;Z)=I(Y;Z\mid X)\ge 0$. Both signs follow from the nonnegativity of conditional mutual information; the first gap also gives equality in data processing inequality.

Let $A,B,Z$ be pairwise disjoint sets of vertices in a directed acyclic graph. A graph path, traversed without requiring its edges to point in the direction of travel, is active given $Z$ precisely when every interior noncollider lies outside $Z$ and every collider either belongs to $Z$ or has a descendant in a directed graph in $Z$. The sets $A,B$ are D-separated by $Z$ when no active graph path connects them.^[ref]

In the three-vertex chain $X\to M\to Y$ and fork $X\leftarrow M\to Y$, conditioning on $M$ blocks the only graph path. In $X\to M\leftarrow Y$, the collider blocks the graph path before conditioning; conditioning on $M$ or its descendant in a directed graph activates it. Other vertices can still block a longer graph path.

Under the global Markov property of a directed acyclic graph, D-separation implies conditional independence. The reverse implication for a particular probability distribution requires faithfulness of a directed acyclic graph: additional conditional independences can arise through special parameter choices.

Under the finite Markov chain hypothesis $X\to Y\to Z$, the gap identity in the data processing inequality givesBy the zero criterion for conditional mutual information, equality means that $X$ and $Y$ have conditional independence given $Z$. After seeing the processed output, the original observation provides no additional information about $X$. Similarly, equality in the other bound, $I(X;Z)=I(Y;Z)$, holds exactly when $I(Y;Z\mid X)=0$.

Equality does not require retaining every detail of $Y$. Let $X,W$ be independent fair bits, put $Y=(X,W)$, and let the processed output be $Z=X$, the first coordinate of $Y$. This is a Markov chain, andAlso $I(X;Y\mid Z)=H(X\mid X)-H(X\mid X,W)=0$. The output discards the entire unrelated bit $W$, reducing $H(Y)=2$ to $H(Z)=1$, while preserving all mutual information about $X$. This is what makes equality useful for choosing a compact observation that retains all information about the quantity of interest.

Eugene Katsevich works on high-dimensional statistics, variable selection and multiple testing, motivated in part by genetics and genomics. These problems require distinguishing reproducible signals from the many associations that can arise when thousands of features are examined.^[ref]

His joint work on the conditional randomization test uses a known feature conditional distribution to calibrate tests of conditional independence. Distillation makes flexible prediction methods more practical as test statistics.^[ref]

For a common success chance with beta prior and conditionally independent binomial observations of sizes $m_i$, put $M={\sum }_i{m}_i$ and $s={\sum }_i{x}_i$. Beta-binomial conjugacy gives posterior mean $(\alpha +s)/(\alpha +\beta +M)$, equal to $Z(s/M)+(1-Z)\alpha /(\alpha +\beta )$. Multiplying by the next exposure predicts its expected count. The factor tends to one as exposure increases and to zero as prior concentration increases at fixed prior mean.

A Gamma–Poisson hierarchical model uses $Y_i\mid {\theta }_i\sim \mathrm{Poisson}(t_i{\theta }_i)$, ${\theta }_i\mid b\sim \mathrm{Gamma}(a,b)$, and $b\sim \mathrm{Gamma}(c,r)$, with gamma distributions in shape-rate convention. Conditional independence gives full conditional distributions ${\theta }_i\mid b,y\sim \mathrm{Gamma}(a+y_i,b+t_i)$ and $b\mid \theta ,y\sim \mathrm{Gamma}(c+na,r+{\sum }_i{\theta }_i)$.