Bayes' theorem Created 2026-09-24 Updated 2026-10-14
For a finite measurable partition with positive probabilities, and an event of positive probability,Its beauty is that a model predicting observations from a hidden state can also describe uncertainty about the state after an observation. Each numerator is a joint probability; the denominator, supplied by the law of total probability, makes the answers sum to one. This is a direct consequence of conditional probability, rather than an extra axiom.
Select the coin from law of total probability once, then toss that same coin repeatedly, with conditionally independent tosses given its identity. Let denote the -head coin, initially of prior probability . After one head , its posterior probability isThe likelihood of a head is larger under , but remains less likely than the other coin because its prior probability was smaller. A large likelihood for an observation does not itself imply a large posterior probability for the hypothesis.
After an ordered sequence containing heads and tails, its likelihood ratio between the two coins is . Bayes' theorem therefore gives posterior oddsEach head multiplies the odds by and each tail by . Three heads give posterior probability , still below one half; four give , above one half. The finite-model update and the distinction between prior probability, likelihood and posterior probability are set out in MIT 18.05, Reading 11: Bayesian Updating with Discrete Priors.
Updating uncertainty about one selected coin after repeated observations
. Details. The hidden selection also creates dependence. Each toss has marginal head probability , but two heads have joint probability , greater than . Their indicator covariance is . Seeing a head makes the head-favoring coin more likely, so it also raises the chance of a future head. This is covariance induced by a shared latent variable, despite conditional independence of the tosses given the coin.
Historically, Thomas Bayes's essay on an unknown success probability appeared in 1763 after his death, communicated by Richard Price. Its opening problem asks how observed successes and failures constrain that unknown probability; Price's introductory letter explains the connection to reasoning from experience. That is a more substantial inference problem than simply rearranging today's elementary identity. [ref]
https://www.gtfp.cs.rhul.ac.uk/pulskamp/Bayes/bayesessay-rjp.pdf. Bayes, An Essay towards solving a Problem in the Doctrine of Chances, 1763; original text transcribed, introductory letter and statement of the problem
Let be fixed and known, and let be independent and identically distributed Bernoulli random variables with common success probability . Their joint probability mass function, on the binary sample space , isThere are arrangements with successes, so has the binomial distribution. For and , its exact conditional full-data probability mass function isThis conditional distribution is uniform over arrangements, with no dependence on : the sum is a sufficient statistic. [ref]
Sufficiency (Stat 210a, Fall 2025) by Will Fithian. Sufficiency, Will Fithian, “Independent Bernoulli sequence”: conditional pmf
Four Bernoulli observations grouped by their sufficient sum
. All sixteen binary sequences are grouped by . Within each group the conditional distribution is uniform; changing changes the group's total probability, not the conditional weights within it. Details.For , the group sizes are and the within-group conditional probabilities are respectively . For example, leaves exactly , each with conditional probability . At , each such sequence has unconditional probability and the group has probability . At , these values are and ; at they return to and . The conditional weights stay in all three cases.
At the boundary , only has positive probability; at , only does. Dividing by on the other groups would divide by zero. Nevertheless, the same uniform-on-arrangements Markov kernel is a valid common version for the entire family : it gives the correct point mass on the one observed group at each boundary and assigns permissible versions on its unobserved groups. Define the joint probability mass function there by the original product of the one-trial Bernoulli probability mass functions, or use the limiting convention in the compact formula.
The sample mean is an unbiased estimator of and retains the sufficient statistic because is known. By contrast, is also an unbiased estimator of but, for , is not sufficient: the conditional success probability of the independent given is still . Thus unbiasedness does not imply sufficiency.
Sufficiency also does not imply completeness of a statistic. For , the full-data sufficient statistic admits the nonzero function with expectation zero for every . Nor does minimality imply completeness of a statistic: for and the restricted statistical model , is a minimal sufficient statistic since the likelihood ratio distinguishes its three values. Yet has expectation at both parameter values and is never zero. Changing the parameter family can change completeness of a statistic even while preserving sufficiency.
Binary Brownian drift filter 2026-10-14
For with an independent equiprobable drift , the likelihood ratio between the two drifts is . Multiplying the normal distribution densities of successive increments proves this ratio for every observation partition; continuity then gives the full observed-path posterior. The Bayes' theorem gives . Thus the observed drift is . The process is an observed-filtration martingale with quadratic variation , so the Lévy characterization of Brownian motion makes it a Brownian motion. The observed state is a scaled diffusion with hyperbolic tangent drift.
Let be the Bernoulli distribution with success probability and the Bernoulli distribution with success probability . For ,The two summands have product . Their arithmetic mean is at least their geometric mean, with equality at . ThusThe same argument gives for complementary parameters . At the laws agree and .
With equal prior probabilities, count the successes among observations. The likelihood ratio is , so choose when and when . If is even, either decision or a fair tie randomization gives the same average error at . Under , has the binomial distribution with parameters . Symmetry therefore gives the exact optimal average errorThe last term applies only for even ; it represents half the tie probability. Direct finite sums give , and . Twenty observations reduce the optimal average error to about 0.158%. The finite-sample Chernoff error bound at is , a guarantee rather than the exact error.
Exact repeated-coin error, the finite-sample Chernoff error bound, and the limiting error exponent
. Original finite-binomial calculations under two fixed independent coin laws with equal priors. The first panel uses a logarithmic scale. The normalized negative log error approaches ; finite-sample error need not equal the simple exponential. Details.Reducing the exact error from to illustrates the value of repeated evidence. The example assumes a single fixed coin law throughout the experiment. If repeated sensor readings share an unmodelled common error, increasing their number need not supply this much independent information.
Conditional diagnosticity in attribution 2026-10-25
Behavior is evidence for a disposition , given a situation , to the extent that its conditional likelihood ratio differs from one. By Bayes' theorem this ratio multiplies the prior odds to give the posterior odds. If a constraint forces the same behavior regardless of the disposition, the ratio is one and the behavior leaves those odds unchanged. This does not exclude diagnostic information in other features, such as wording or voluntary choices. The argument identifies the assumptions needed to call a dispositional attribution biased.
Under local asymptotic normality, a fixed shift has a likelihood ratio limit with . This is positive with mean one. Le Cam first lemma gives mutual contiguity of the product laws, so consistency at the central parameter transfers to these local alternatives.
Egon Pearson 2026-11-05
Egon Pearson developed the theory of statistical hypothesis testing with Jerzy Neyman. Their framework distinguishes controlling type I error under a null hypothesis from maximizing statistical power against a specified alternative hypothesis. University College London's institutional history identifies this collaboration as a central contribution.[ref]
https://www.ucl.ac.uk/mathematical-physical-sciences/statistics-old/department-information/our-early-history. Departmental history, Egon Pearson and collaboration with Neyman
The Neyman-Pearson lemma makes the tradeoff precise for two simple hypotheses with densities and . At a fixed allowed type I error, the most powerful test rejects where the likelihood ratio is largest, with boundary randomization if needed. To see the principle, compare its test function with another admissible : the product is nonpositive pointwise. Integrating and using the size constraint shows that cannot have greater statistical power.
Exponential tilting Created 2026-09-24 Updated 2026-11-05
exponential tilting reweights a probability law in favour of larger or smaller values. Let be finite at the chosen parameter. Define the tilted law by its Radon-Nikodym derivativeIts total mass is one by construction. Positive favours larger , while negative favours smaller ; it changes probabilities without creating values outside the original support.
For in the interior of the finite moment-generating-function domain, differentiation under the integral givesThe interior-domain condition supplies the integrable exponential bounds needed for these derivatives. Endpoint tilts can exist without a finite tilted mean, so existence of a normalized tilt alone is insufficient for the mean formula.
For a Bernoulli distribution with success probability , the tilted success probability is : tilting adds to its log odds. A Normal distribution tilts to by completing the square. These calculations explain why exponential tilting is useful for making a rare Sample mean typical while retaining an explicit Likelihood ratio.
Failure of sign recovery from magnitude-only calibration data occurs when opposite signed calibration models induce exactly the same distribution of every recorded magnitude. Under this symmetry, more repetitions can estimate the magnitude distribution accurately without deciding the missing sign. This is the limitation relevant to sign-sensitive measurements that resolve a covariance-axis ambiguity.
Let with sign , known and symmetric noise having density . If the recorded value is , then for Symmetry gives . Independent repetitions multiply identical densities, so their likelihood ratio remains one. With equal sign priors and equal decision costs, no decision rule based solely on these magnitudes can have average sign-error probability below1/2.
For normal distribution noise with standard deviation , the magnitude has a folded normal distribution:where is the standard normal density. The formula depends on , not its sign. The pinned numerical reference provides the equivalent standardized folded-normal density and its scale transformation.[ref]
SciPy 1.16.2 documentation. scipy.stats.foldnorm, Notes, density and loc/scale convention
For the original calibration scale and , recording a signed value can distinguish the two models strongly, but recording only its magnitude cannot do so under the symmetric model. This is an information loss in the measurement map, rather than a shortage of averaging trials.
The symmetry assumption is essential. As an original counterexample, let noise equal−1 with probability2/3 and+2 with probability1/3, giving zero mean but an asymmetric distribution. With , the positive-sign magnitudes are0 or3, while the negative-sign magnitudes are2 or1. Their disjoint supports permit sign recovery if this asymmetric noise model is independently established. The calibrated asymmetry itself supplies sign-reference information absent from the symmetric case.
Similarly, a genuinely signed projection or a known phase reference can break the original ambiguity. An unverified readout polarity or unknown amplitude sign can preserve it. Specify which signed information the apparatus actually records before interpreting a magnitude fit as an oriented calibration.
Fix ravens and nonblack objects in a finite population. Under , all ravens are black; under , exactly one is nonblack. The total numbers are held fixed under both hypotheses, so the classification of one nonraven also changes between hypotheses. This is a deliberately specified model for Bayesian confirmation, not a universal rule about how bird observations supply evidence.
A uniform draw among nonblack objects returns a nonraven with probabilities under and under . A uniform draw among ravens returns a black object with probabilities and . The respective likelihood ratios for over are therefore and . When , the former is closer to one, so supplies less support under these assumptions.
With equal prior probabilities, Bayes' theorem gives posterior probabilities and . More generally, with prior probability for , replace the two expressions by and . Previously known information or a different sampling protocol requires different likelihoods; selecting an object already known to be a nonraven need not supply new evidence.
Fixed-mean normal variance test 2026-10-31
For independent observations, testing a fixed variance against larger variances uses the sum of squares . The likelihood ratio for any simple larger variance is strictly increasing in . The Neyman-Pearson lemma therefore gives the displayed rejection rule, where is the chi-squared distribution function with degrees of freedom. Its threshold is independent of the selected larger variance, so it is a uniformly most powerful test for the one-sided composite alternative. Its power function is , increasing continuously from zero to one and equaling at the null. A known nonzero mean is handled by replacing each observation by its difference from that mean.
Gaussian-noise posterior for an autologistic image Created 2026-10-07 Updated 2026-11-05
For an autologistic binary-image model on a finite undirected graph, let and give the configuration prior mass proportional toSuppose are independent Gaussian distribution observations with expected value and unit variance. Their local log-likelihood ratio for versus isWriting , the full posterior conditional probability is thereforeThus the local evidence changes the one-site field while retaining the same neighbor interaction. For a noise variance , replace by .
The graphical conditional independence survives this update: vertices in any independent color class are conditionally independent given the other vertices and the observations. On a nearest-neighbor square grid, the two checkerboard colors give such classes, so one color can be updated in parallel in a Gibbs sampler. An arbitrary graph need not be bipartite and may require more colors. Updating every vertex simultaneously from its old neighbors is not the same as this exact conditional block update.
Genetic-study ascertainment 2026-10-13
Ascertainment in a genetic study specifies how individuals or families enter a genetic study. If is the event of inclusion, then . Selection on disease phenotype can change an unconditional pedigree likelihood. When comparing genetic marker probabilities conditional on a specified disease pattern, selection depending only on that pattern cancels from the linkage likelihood ratio.
The bridge from detector readout to a particle resonance is four-momentum conservation. In units , sum the measured final-state four-momenta:For two photons separated by angle , gives the displayed diphoton formula. Two back-to-back photons of each have . A four-lepton candidate instead uses four four-momenta in the same sum. Collision debris outside the selected decay is not included. Since the partons carry unknown fractions of the incoming proton momenta, reconstructing the decay products is more informative than assigning the full beam energy to each candidate.
Suppose the intrinsic decay width is negligible compared with the stipulated detector spread. Let the reconstructed signal mass follow a normal distribution with mean and standard deviation . For a total expected selected signal and flat background density events per electronvolts, the expected count density isThis is a deliberately chosen response model, not a fit to either experiment. In the fixed window with , direct integration givesThe expected signal counts are for and for . Doubling the spread lowers the fraction in the same window without changing the total signal. Detector resolution can therefore change the visibility of a peak even with identical particle production.
As a separate counting example, stipulate an observed integer count and an exactly known background mean . The Poisson distribution gives a likelihood for nonnegative signal mean . Its maximum-likelihood estimator is , and the likelihood ratio statistic against isThis tests a specified window with a specified background, and is not either experiment's discovery significance. If the background mean is allowed to vary freely, background alone can choose and fit that count exactly. Control samples, a defensible mass-shape model, nuisance parameters and corrections for searching many masses are therefore essential to the experimental inference.
Kullback-Leibler divergence Created 2026-09-24 Updated 2026-10-14
The Kullback-Leibler divergence, also called relative entropy, compares two probability distributions in a specified order. In , observations follow and the comparison probability distribution is : the first argument supplies the averaging weights. For probability mass functions on a finite alphabet,Here is a random variable with probability distribution . For probability density functions relative to the same measure , replace the sum by . These formulas use natural logarithms unless stated otherwise. The expected value is under , even though appears in the denominator. [ref]
Take and on two outcomes. Direct calculation givesThe second outcome contributes negatively to the first sum: assigning it more probability than does makes its individual contribution negative. Nevertheless, relative entropy nonnegativity guarantees a nonnegative total. Reversing the arguments changes both the likelihood ratio and the averaging probabilities; taking an absolute value inside the logarithm would define a different quantity.
The units depend on the logarithm: natural logarithms give nats, whereas base-two logarithms give bits. Divide a value in nats by to obtain bits; the forward example is approximately bits. Changing the base rescales every Kullback-Leibler divergence by the same positive factor. Since the two calculated directions differ, Kullback-Leibler divergence is asymmetric and cannot be a metric. It also need not satisfy the triangle inequality. [ref]
https://web.stanford.edu/~kedart/ee376a_winter1617/Lecture_3.pdf. Tse, relative entropy, 2017, Section 3.3
The operational meaning is an expected penalty for using the wrong probability distribution. A prediction incurs log-loss when outcome occurs, the negative of the logarithmic scoring rule. Under , its expected value is the cross-entropy . Predicting with itself incurs information entropy , soThus Kullback-Leibler divergence is exactly excess expected log-loss, whenever these separate quantities are finite. [ref]
https://web.stanford.edu/class/ee276/files/previous_notes/lecture_2.pdf. Weissman, information measures, 2024, property 3
For lossless source coding, ideal codeword lengths similarly give excess expected length in bits per symbol. Actual prefix codes require integer codeword lengths; Shannon's source coding theorem explains how long blocks approach ideal per-symbol lengths. The interpretation concerns an average, not a fractional physical bit attached to every outcome. In our example, ideal coding with costs one bit per symbol, while bits; the difference is exactly the computed bits. [ref]
https://web.stanford.edu/~kedart/ee376a_winter1617/Lecture_3.pdf. Tse, information theory, 2017, Section 3.4
Zeros require explicit conventions. An outcome with contributes zero, including when ; the convention agrees with the relevant limit. If but , the Kullback-Leibler divergence is infinite: the predictor has declared impossible an outcome that can occur. On a finite alphabet, the probability support of must therefore be contained in the probability support of for a finite answer. For example, and give , but . For general probability measures, failure of absolute continuity of measures likewise gives infinity. Even does not guarantee a finite integral. [ref]
https://arxiv.org/pdf/1206.2459. van Erven and Harremoës, Rényi Divergence and Kullback-Leibler Divergence, 2014, Section II
To see how confidently wrong predictions are punished, hold the true Bernoulli distribution parameter at and vary the predicted parameter . Its binary relative entropy isIt vanishes at , equals approximately nats at , and equals nats at . Both endpoints give infinity because both outcomes remain possible under . Predicting almost no successes is especially costly because successes actually occur three quarters of the time. The reversed binary relative entropy stays finite at the endpoints: its averaging probability distribution concentrates on just one outcome, and assigns positive probability to both.
For model selection, minimizing expected log-loss ranks candidate statistical models by , since is common to them. The true probability distribution is usually unknown, so one estimates predictive performance using fresh observations or cross-validation. Fitting and assessing on the same data can reward overfitting; a small fitted Kullback-Leibler divergence alone does not establish useful prediction.
In variational inference, a tractable trial probability distribution commonly minimizes against a target posterior distribution . Maximizing the evidence lower bound accomplishes this without evaluating the target's normalization constant. The averaging now follows : placing trial probability where the posterior distribution assigns almost none is strongly penalized. The chosen direction and trial family therefore matter. [ref]
https://www.cs.columbia.edu/~blei/papers/BleiKucukelbirMcAuliffe2017.pdf. Blei, Kucukelbir and McAuliffe, Variational Inference: A Review for Statisticians, 2017, Section 2.2
These comparisons do not by themselves establish causal inference. Two statistical models can reproduce the same observed probability distribution while predicting different responses to an intervention. Choosing the smaller Kullback-Leibler divergence therefore needs additional assumptions or experimental evidence before supporting a causal claim. [ref]
https://ftp.cs.ucla.edu/pub/stat_ser/r350.pdf. Pearl, Causal inference in statistics: An overview, 2009, Section 2
The original reference is S. Kullback and R. A. Leibler's 1951 paper [ref]
https://doi.org/10.1214/aoms/1177729694. On Information and Sufficiency
, published in The Annals of Mathematical Statistics 22(1), pages 79–86. Its title connects this comparison with sufficient statistics: the historical setting was statistical information, before today's widespread applications in prediction and variational inference. Likelihood function Created 2026-09-24 Updated 2026-10-14
For a fixed observed sample , a likelihood function iswhere is the probability mass function or probability density function of the observations under the statistical parameter . All densities must be defined relative to the same parameter-independent dominating measure. The sample is fixed; the parameter is the function's argument. With independent and identically distributed random variables, the joint mass or density factors as . This definition and the distinction between sampling probability and parameter uncertainty are explained in MIT 18.05, Reading 10b, sections 3 and 3.2.
For an observed ordered sequence of ten independent Bernoulli trials containing eight successes and two failures, each success contributes and each failure contributes , givingIf the observation is instead only the total count , its binomial distribution probability mass function sums over the possible ordered sequences:The raw heights differ: at they are approximately and . Their maximum-likelihood estimates are identical. More generally, multiplication by any finite positive constant independent of leaves the maximizing parameter values and all likelihood ratios unchanged. Multiplication by a parameter-dependent factor can change the MLE; dropping such a factor is not justified. Relative likelihood height sets the peak to one, without making the area under the curve one.
Normalization of is over possible data values for each fixed parameter. It imposes no requirement that . For example, the ordered Bernoulli trials likelihood above hasIts integral is not one, and its values are not posterior probabilities for . When the Bayesian posterior is defined, Bayes' theorem multiplies the likelihood by a prior density and normalizes that product. Choosing to normalize a likelihood function alone is an additional choice, such as using a uniform prior in those parameter coordinates, rather than part of the definition of likelihood.
For observations with a probability density function, the probability of an exact sample point is zero. If a scalar density is continuous at the observed value , thenThus the density compares probabilities of small, equal observation intervals at different parameter values; it is not itself the probability of that exact real number. The equal-width factor is independent of the parameter. This gives visual intuition for density likelihoods, as in MIT 18.05, Reading 10b, section 4. The normal distribution example in maximum-likelihood estimator makes the comparison explicit. If only rounded intervals are recorded, their integrated probabilities give the exact observation likelihood; using midpoint densities is an approximation.
Likelihood ratio Created 2026-09-24 Updated 2026-11-05
For two specified model densities or mass functions , the likelihood ratio at observed data is wherever the denominator is positive. It measures relative support for these two models at that observation, not a posterior probability without prior odds. One Bernoulli success gives ratio for success probabilities and ; a failure gives ratio . For independent observations the ratios multiply. In a likelihood-ratio test of composite models, optimized likelihoods replace the two fixed densities, with an explicitly stated numerator convention.
Let for independent and identically distributed observations, with having an exponential distribution of unknown scale . Test the null hypothesis within . For nondegenerate data, the likelihood function isMaximizing over the scale gives , and the resulting profile likelihood increases with . Thus the unrestricted maximum-likelihood estimates are and , while the null estimate is . The likelihood ratio is consequently
Under the null, put and . The memoryless property gives independent gamma distributions of shapes and and common scale . For a direct verification, fix which observation is the minimum and write its value as and the other observations as . Their joint density is proportional to , which factors into an exponential density for the minimum and independent exponential densities for the excesses. The minimum's index does not affect the resulting laws of .
By beta sampling by gamma ratios, . ThereforeAn exact test size is obtained by rejecting for . Equivalently,Although the alternative adds one endpoint parameter, a regular one-parameter Wilks theorem calibration is incorrect: the support depends on the endpoint, and the null lies on its boundary. In particular, tends to a chi-squared distribution with two statistical degrees of freedom, not one.
Likelihood-ratio test for a normal variance 2026-10-25
For an independent normal distribution sample with unknown mean, test against unrestricted positive variance. Put and assume . Profiling the mean and variance gives the likelihood ratio for a nondegenerate sample. Its logarithmic derivative changes sign at , so small ratios select a lower and an upper tail.
Under the null hypothesis, Cochran's theorem gives . The exact size- cutoffs therefore satisfyThe second equation makes the two boundary ratios equal. Rejection on or is consequently the exact likelihood-ratio test, and its tails need not have equal probability. For a large sample, the Wilks theorem instead calibrates by . A Bartlett correction for a normal variance with unknown mean improves this asymptotic calibration; it is unnecessary when using the exact distribution.
Take two independent statistical samples from beta distributions with parameters and , of positive sizes . Put , , and . To test the null hypothesis , divide the maximized common-rate likelihood function by the unrestricted maximum to obtain the displayed likelihood ratio. Indeed the unrestricted fitted rates are , and the common fitted rate is ; substitution cancels the exponential factors.
Under the null hypothesis, and have independent gamma distributions with a common rate. Beta-gamma independence gives . The derivative of changes sign at , so the exact critical region of size is or , with . Its cutoffs satisfyHere is the beta distribution cumulative distribution function. Equal likelihood ratios at the cutoffs do not generally mean equal tail probabilities. Increasing the ratio threshold continuously adds positive null distribution mass on both sides, proving uniqueness for .






