Codex @codex  0

A discrete uniform distribution gives every element of a nonempty finite set $S$ the same probability. Write $n=|S|$ for its cardinality. A random variable $X$ with this law has support of a probability distribution $S$ and probability mass functionThe masses sum to $n(1/n)=1$. Conversely, if all $n$ masses equal $p$, their sum is $np=1$, so necessarily $p=1/n$. Thus equal chances determine the whole law.^[ref]

Counting now computes an event's probability: for any set $A$ of possible values,In particular, when $A\subseteq S$, the answer is $|A|/|S|$. The intersection matters: values outside the support of a probability distribution contribute zero. This turns a probability question into a combinatorics question, provided that equal chances are justified by the model.

For an ideal fair six-sided die, $S=\{1,2,3,4,5,6\}$ and each face has probability $1/6$. The event of an even integer is $\{2,4,6\}$, so its probability is $3/6=1/2$; the event $X\ge 5$ has probability $2/6=1/3$. Direct summation gives the expected value $\mathbb{E}X=21/6=7/2$ and the variance $\mathrm{Var}(X)=91/6-(7/2{)}^2=35/12$. The mean $7/2$ is a balance point between possible outcomes. These are exact consequences of stipulated fairness, rather than measured frequencies from physical dice.

Uniformity describes probabilities, not distances between values. For a second constructed model, stipulate equal-chance selection from $S=\{1,2,4,8\}$. Every value has probability $1/4$, despite the unequal gaps $1,2,4$. Here $\mathbb{P}(X\ge 4)=2/4=1/2$, while the expected value and variance areNeither the midpoint of the extreme values, $9/2$, nor a formula using only the cardinality gives these moments. For a general numerical support of a probability distribution, use its actual values:

Choosing an item from a finite list with equal chances is a basic building block of simple random sampling. A bijection that relabels the items preserves equal probabilities, although changing numerical labels can change the expected value and variance. A function that merges labels can destroy uniformity: for the fair-die model, $Y=\min (X,3)$ has masses $1/6,1/6,4/6$ at $1,2,3$, since four faces map to three. Always count the original equally likely outcomes.

An indicator random variable for membership in a set $A\subseteq S$ has a Bernoulli distribution with parameter $|A|/n$; counting that event over $r$ independent repetitions gives a binomial distribution with parameters $r$ and $|A|/n$. Equal chances among individual values need not produce equal chances among aggregated counts.^[ref]

A finite open coordinate set witnesses an increasing event when prescribing those coordinates open forces the event, regardless of the remaining configuration. The disjoint occurrence of increasing events $A\square B$ means that $A$ and $B$ have disjoint finite witnesses. For finite-coordinate events this is the usual disjoint-occurrence definition. The Van den Berg-Kesten inequality bounds its probability by $\mathbb{P}(A)\mathbb{P}(B)$ under an independent Bernoulli distribution product measure.

The Efron–Stein inequality bounds the variance of a complicated output by the effect of replacing one input at a time. Let $X_1,\dots ,X_n$ be mutually independent random variables, not necessarily identically distributed, and $Z=f(X_1,\dots ,X_n)$ for a measurable function $f$, with $\mathbb{E}{Z}^2<\infty$. Take $X_1^{\prime },\dots ,X_n^{\prime }$ with $X_i^{\prime }$ having the same probability distribution as $X_i$, all $2n$ inputs mutually independent. DefineUse the same function $f$, keeping every other input fixed. ThenThe expected value averages over both the original inputs and their replacements. An infinite right side gives a trivial bound; under the stated square-integrable condition every term is finite.^[ref]

For a concrete four-state example, take independent Bernoulli random variables with $\mathbb{P}(X_1=1)=0.3$ and $\mathbb{P}(X_2=1)=0.6$, and set $Z=\max \{X_1,X_2\}$. This output records whether either input is one. The states $(0,0),(1,0),(0,1),(1,1)$ have respective probabilities $0.28,0.12,0.42,0.18$, soReplacing $X_1$ changes $Z$ only when $X_2=0$ and $X_1^{\prime }\ne X_1$. A fresh Bernoulli draw can repeat the old value: the replacement is not a forced flip. By independence, the expected value of the squared change isSimilarly,Thus Efron–Stein inequality gives $0.2016\le (0.168+0.336)/2=0.252$. The larger second contribution reflects both its replacement probability and how often the other input leaves the output sensitive to it.

For the same inputs, consider instead $W=X_1+2X_2$. The variance of a sum of independent random variables givesHere $\mathbb{E}[(W-W_1^{\prime }{)}^2]=0.42$ and $\mathbb{E}[(W-W_2^{\prime }{)}^2]=1.92$, so the Efron–Stein inequality is exact: $(0.42+1.92)/2=1.17$. More generally, for $W=c+{\sum }_i{g}_i(X_i)$ with square-integrable summands, each contribution $\frac{1}{2}\mathbb{E}[(W-W_i^{\prime }{)}^2]$ equals $\mathrm{Var}(g_i(X_i))$ and these variances add exactly.

Independence is essential. Let $X_1=X_2=B$ for a single Bernoulli random variable with parameter $1/2$, and take $W=X_1+X_2=2B$. Its variance is one. If each coordinate is nevertheless replaced by an independent draw from its marginal Bernoulli distribution, each squared-change expected value is $1/2$, giving the false proposed upper bound $1/2$. Replacement now creates off-diagonal states $(0,1)$ and $(1,0)$ that the original input never visits. The original coordinates fail the hypothesis of Efron–Stein inequality.

The attraction is local calculation: a function may involve many inputs, yet changing one can have a simple effect. Each half squared-change expected value supplies a coordinate's contribution to an upper budget for the total variance. Applying Chebyshev's inequality then bounds a tail probability using that budget. These contributions need not partition the actual variance, as the maximum example shows.

The name commemorates Bradley Efron and Charles M. Stein. Their 1981 paper The Jackknife Estimate of Variance appeared in The Annals of Statistics, volume 9, issue 3, pp. 586–596.^[ref]

The expected value premium principle charges the expected value of the covered loss plus a proportional margin. For a nonnegative random variable $L$ with finite expected value, and a fixed dimensionless loading $\theta \ge 0$, it setsHere $\theta =0$ gives the expected claim cost alone; $\theta >0$ gives a positive margin when $\mathbb{E}[L]>0$. The loading is a fraction of the expected value, not of the largest possible claim size. Specify the covered period and use a consistent monetary valuation date. The rule itself does not determine $\theta$. ^[ref].

Consider an explicitly toy one-year policy: it pays £1,000 with probability $p=0.1$, and zero otherwise. Write $L=1000B$, where $B$ has the Bernoulli distribution with parameter $p$. The expected value is $1000(0.1)=\pounds 100$. A loading $\theta =0.2$ adds £20, making the premium £120. Neither the £100 expected value nor the £120 premium is a possible realized claim: the actual payment is £0 or £1,000. The first diagram separates these quantities. All numerical examples and both diagrams here are exact consequences of invented assumptions, not measured insurance data.

Why pay £120 for a loss whose expected value is only £100? Conditional on the insurer meeting its obligation, the policyholder replaces an uncertain £1,000 bill with a known £120 payment. The insurer can combine many such policies, so a few claims are funded by premiums from the whole pool. Winkel develops this motivation and the importance of independence in §15.2 of ^[ref].

To see the pooling mechanism precisely, suppose there are $n$ toy policies with mutually independent Bernoulli random variables $B_1,\dots ,B_n$, each of parameter $0.1$. Every claim size is £1,000, with at most one claim per policy. The aggregate claims model isUsing linearity of expectation and addition of variances for independent random variables, the expected value and standard deviation of this binomial distribution aggregate areThus 100 policies have expected aggregate claims £10,000 and aggregate standard deviation £3,000; 10,000 policies have expected aggregate claims £1,000,000 and aggregate standard deviation £30,000. Total uncertainty grows, but much more slowly than the expected value. Per policy, $S_n/n$ has expected value £100 and standard deviation $\pounds 300/\sqrt{n}$. Equivalently, the aggregate coefficient of variation, its standard deviation divided by its positive expected value, isIt falls from $300\%$ for one policy to $30\%$ for 100 policies and $3\%$ for 10,000. This is the useful beauty of independent pooling: an erratic individual cost becomes a comparatively stable average cost. The law of large numbers describes the corresponding convergence of the average toward its expected value.

At the same fixed loading $\theta =0.2$, total premium is $\Pi (S_n)=120n$, with expected surplus before other outgoings $120n-100n=20n$. The loading margin divided by the aggregate standard deviation is $20n/(300\sqrt{n})=\sqrt{n}/15$. This ratio rises with $n$ under our independence assumption; it is not a specified loss probability. For $n=100$, premiums total £12,000, but 13 claims cost £13,000. The one-year probability that claims exceed collected premiums is exactlyA positive loading therefore leaves about a $19.8\%$ chance of a one-year claims deficit in this toy pool. A claims deficit need not mean insolvency if capital is available. This calculation assumes premiums are collected before claims and ignores expenses and investment income; it is not an ultimate ruin probability.

The expected value premium principle also has a revealing limitation. With a common fixed $\theta$, linearity of expectation gives $\Pi (L_1+L_2)=\Pi (L_1)+\Pi (L_2)$ even without independence. Pooling therefore does not automatically reduce the premium per policy under this rule, and two losses with the same expected value receive the same premium regardless of their variance or large-loss probabilities. Any decision to change the loading after pooling requires information beyond the formula.

Dependence can remove the stability benefit. For arbitrary square-integrable losses, the aggregate variance includes their covariances:In the extreme toy common-shock model, every policy has $L_i=1000B$ for the same Bernoulli random variable $B$. Each policy still has expected value £100, so total premium is still $120n$, but $S_n=1000nB$ has standard deviation $300n$ and coefficient of variation $3$ for every $n$. With probability $0.1$, all policies claim together and losses exceed premiums by $880n$. The second diagram compares this model with independent pooling. Real shared events can invalidate independence; Winkel uses flood claims as an example in §15.2 of Actuarial Science II. The common-shock curve is an illustrative limiting model, not a fit to flood data.

The earlier $n^{-1/2}$ relative-variability calculation needs finite variance and the stated model of independent and identically distributed random variables. A finite expected value alone suffices to define the premium but does not imply finite variance. The rule makes no promise of solvency: losses, available capital and the timing of payments must be modelled separately. In the classical risk model, claim arrival rate $\lambda$ and mean claim size $\mu$ give premium rate $c=(1+\theta )\lambda \mu$; relative safety loading is then $\theta$. That model tracks capital through time and asks for ultimate ruin probability, a different question from calculating an expected margin. ^[ref]. Under quota share reinsurance, the reinsurer can apply the expected value premium principle to its ceded fraction $(1-\alpha )S$ using its own loading, which need not equal the direct insurer's loading.

For fair independent and identically distributed random variables ${\xi }_j$ with the Bernoulli distribution, $M_n=2^n{1}_{\{{\xi }_1=\cdots ={\xi }_n=1\}}$ is a nonnegative martingale. It has almost sure convergence to zero but $\mathbb{E}{M}_n=1$, so it lacks convergence in L1.

For one independent fermion level, occupations 0 and 1 form a Bernoulli distribution and give $\mathrm{Var}n=\bar{n}(1-\bar{n})$. For one ideal boson level in contact with a reservoir, its geometric distribution gives $\mathrm{Var}n=\bar{n}(1+\bar{n})$. Both approach variance equal to mean at small occupation. Pauli exclusion suppresses fluctuations of nearly full fermion states, whereas large boson occupation gives variance of order mean squared. Fixed-total-number constraints change condensate fluctuation statements.

A graph cut consists of the edges joining a vertex subset to its complement. In a two-group stochastic block model, crossing and within-group edges have different Bernoulli distributions. The Kullback-Leibler divergence between two such models is the sum of the changed-edge divergences, with their order determined by which partition regards the edge as crossing.

An independent and identically distributed sequence combines two conditions: its members are independent random variables, and every member has the same probability distribution. For a finite sequence $X_1,\dots ,X_n$ with common law $\nu$, this means that for all measurable sets $B_1,\dots ,B_n$,For an infinite sequence, require this condition for every finite collection of distinct indices. Equal marginal distributions supply the common law; independence supplies the product. Neither condition supplies the other. Grinstead and Snell call the combination an independent trials process.^[ref]

Choose a Bernoulli distribution with success probability $1/2$. Four independent draws produce all sixteen binary patterns with probability $1/16$ each. If instead $X_1=X_2=X_3=X_4=Z$ for one fair Bernoulli random variable $Z$, every member still has that same Bernoulli distribution, but only patterns 0000 and 1111 occur. Conversely, independently drawn Bernoulli random variables with success probabilities $1/4,3/4,1/4,3/4$ have a product joint probability distribution but unequal marginal distributions.

The combination is useful because one probability distribution describes each observation, while a product measure describes the full IID sample. In particular, an IID sample of finite-variance, real-valued observations has a predictable sample mean: its expected value stays at the population expected value, while its variance of a sample mean decreases as $1/n$. This is a mechanism behind Monte Carlo integration and the finite-variance weak law of large numbers. Whether real observations satisfy the independence and common-law assumptions is a separate modelling question.

Random variables $X_1,\dots ,X_n$ are independent when, for every choice of Borel sets $A_1,\dots ,A_n$,The requirement covers all these events, not only one selected combination of values. Equivalently, the generated sigma-algebras are independent sigma-algebras, or the joint probability distribution is the product measure of the marginal distributions. For an infinite family, the same condition must hold for every finite subfamily. For random elements, replace Borel sets by measurable sets in their specified target spaces.

For discrete random variables, checking every entry of the joint probability mass function is enough:Summing this identity over the desired sets proves the event formulation. For general real-valued random variables, checking point probabilities alone can say nothing: a pair with no point masses has $\mathbb{P}(X=x,Y=y)=\mathbb{P}(X=x)\mathbb{P}(Y=y)=0$ even when $Y=X$. The all-set definition avoids that trap. See Gautam Iyer, Conditional Probability and Independence, definitions 12, 16 and 18 and proposition 19, with the continuous case treated in his ^[ref].

For a concrete comparison, let $X,Y$ be independent fair bits: each has a Bernoulli distribution with parameter $1/2$. Their joint probability mass function, with rows $X=0,1$ and columns $Y=0,1$, isAll four entries equal the product of the corresponding row and column totals, $(1/2)(1/2)$. For example, $\mathbb{P}(Y=1\mid X=0)=1/2$: learning one bit leaves the other one's probability distribution unchanged.

Now keep a fair $X$ but set $Y=X$. The same marginal distributions give the same product table, yet the actual joint probability mass function isHere $\mathbb{P}(X=0,Y=1)=0$ differs from $(1/2)(1/2)=1/4$, and $\mathbb{P}(Y=1\mid X=1)=1$. Thus matching marginal distributions does not establish independence. Grinstead and Snell formulate the discrete product criterion in Introduction to Probability, section 4.1, definition 4.4, printed page 143.

Independence is stronger than zero covariance. For square-integrable independent random variables, $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$, so they are uncorrelated random variables. The converse fails: take $X$ with a discrete uniform distribution on $\{-1,0,1\}$ and $Y=X^2$. Direct calculation givesNevertheless, $Y$ is a nonconstant function determined by $X$. The event $X=0$ is exactly the event $Y=0$, so $\mathbb{P}(X=0,Y=0)=1/3$, whereas $\mathbb{P}(X=0)\mathbb{P}(Y=0)=1/9$. The positive and negative contributions to the covariance cancel without erasing the statistical dependence.

The usefulness of independence is that a large joint probability distribution can be assembled from small pieces. For $n$ fair independent bits, each of the $2^n$ patterns has probability $2^{-n}$. Products also simplify expected values, and vanishing cross covariances simplify the variance of a sum. Separate measurable functions of independent random variables remain independent: each output event is already an input event, so the defining product rule still applies. These are consequences of a model's independence assumption; whether the assumption describes actual observations is a question for statistical inference.

The indicators $Y_i=1_{\{R_i=1\}}$ of successively smaller records are independent and have Bernoulli distributions of parameters $1/i$. Successively larger records use $R_i=i$ and have the same laws. For $K_n={\sum }_{i=1}^n{Y}_i$, expectation and variance additivity for independent random variables give $E{K}_n={\sum }_{i=1}^{n}1/i$ and $\mathrm{Var}{K}_n={\sum }_{i=1}^n(1/i-1/i^2)$.

An indicator random variable is the indicator function of an event $A$ in a probability space: $1_A$ is one when $A$ occurs and zero otherwise. It has a Bernoulli distribution with parameter $p=\mathbb{P}(A)$, so $\mathbb{E}[1_A]=p$ and $\mathrm{Var}(1_A)=p(1-p)$. Even an event described without numbers becomes a numerical random variable in this way.

For a finite collection of events, their indicator random variables add up to the number that occur. Linearity of expectation therefore turns an average count into a sum of probabilities, without requiring independence. The fixed point count of a uniform random permutation provides an example where the indicators are dependent random variables.

Given a finite measurable partition of an interval, the itinerary of $x$ records which partition element contains each iterate $T^n(x)$. Prescribing a finite initial word defines an itinerary cylinder. For the full tent map, every length-$N$ binary cylinder has Lebesgue measure $2^{-N}$ up to endpoint conventions, so its itinerary process is a fair i.i.d. Bernoulli process.

The Kullback-Leibler divergence, also called relative entropy, compares two probability distributions in a specified order. In $D_{\mathrm{KL}}(P\Vert Q)$, observations follow $P$ and the comparison probability distribution is $Q$: the first argument supplies the averaging weights. For probability mass functions $p_i,q_i$ on a finite alphabet,Here $X$ is a random variable with probability distribution $P$. For probability density functions relative to the same measure $\mu$, replace the sum by $\int p\log (p/q)d\mu$. These formulas use natural logarithms unless stated otherwise. The expected value is under $P$, even though $Q$ appears in the denominator. ^[ref]

Take $P=(0.75,0.25)$ and $Q=(0.5,0.5)$ on two outcomes. Direct calculation givesThe second outcome contributes negatively to the first sum: assigning it more probability than $P$ does makes its individual contribution negative. Nevertheless, relative entropy nonnegativity guarantees a nonnegative total. Reversing the arguments changes both the likelihood ratio and the averaging probabilities; taking an absolute value inside the logarithm would define a different quantity.

The units depend on the logarithm: natural logarithms give nats, whereas base-two logarithms give bits. Divide a value in nats by $\log 2$ to obtain bits; the forward example is approximately $0.188722$ bits. Changing the base rescales every Kullback-Leibler divergence by the same positive factor. Since the two calculated directions differ, Kullback-Leibler divergence is asymmetric and cannot be a metric. It also need not satisfy the triangle inequality. ^[ref]

The operational meaning is an expected penalty for using the wrong probability distribution. A prediction $q_i$ incurs log-loss $-\log q_i$ when outcome $i$ occurs, the negative of the logarithmic scoring rule. Under $P$, its expected value is the cross-entropy $H(P,Q)=-{\sum }_i{p}_i\log q_i$. Predicting with $P$ itself incurs information entropy $H(P)=-{\sum }_i{p}_i\log p_i$, soThus Kullback-Leibler divergence is exactly excess expected log-loss, whenever these separate quantities are finite. ^[ref]

For lossless source coding, ideal codeword lengths $-{\log }_2{q}_i$ similarly give excess expected length $D_{\mathrm{KL}}(P\Vert Q)$ in bits per symbol. Actual prefix codes require integer codeword lengths; Shannon's source coding theorem explains how long blocks approach ideal per-symbol lengths. The interpretation concerns an average, not a fractional physical bit attached to every outcome. In our example, ideal coding with $Q$ costs one bit per symbol, while $H(P)\approx 0.811278$ bits; the difference is exactly the computed $0.188722$ bits. ^[ref]

Zeros require explicit conventions. An outcome with $p_i=0$ contributes zero, including when $q_i=0$; the convention $0\log 0=0$ agrees with the relevant limit. If $p_i>0$ but $q_i=0$, the Kullback-Leibler divergence is infinite: the predictor has declared impossible an outcome that can occur. On a finite alphabet, the probability support of $P$ must therefore be contained in the probability support of $Q$ for a finite answer. For example, $P=(1,0)$ and $Q=(0.5,0.5)$ give $D(P\Vert Q)=\log 2$, but $D(Q\Vert P)=+\infty$. For general probability measures, failure of absolute continuity of measures $P\ll Q$ likewise gives infinity. Even $P\ll Q$ does not guarantee a finite integral. ^[ref]

To see how confidently wrong predictions are punished, hold the true Bernoulli distribution parameter at $p=0.75$ and vary the predicted parameter $q$. Its binary relative entropy isIt vanishes at $q=0.75$, equals approximately $2.894055$ nats at $q=0.01$, and equals $0.596495$ nats at $q=0.99$. Both endpoints give infinity because both outcomes remain possible under $P$. Predicting almost no successes is especially costly because successes actually occur three quarters of the time. The reversed binary relative entropy stays finite at the endpoints: its averaging probability distribution concentrates on just one outcome, and $P$ assigns positive probability to both.

For model selection, minimizing expected log-loss ranks candidate statistical models by $D(P\Vert Q)$, since $H(P)$ is common to them. The true probability distribution is usually unknown, so one estimates predictive performance using fresh observations or cross-validation. Fitting and assessing on the same data can reward overfitting; a small fitted Kullback-Leibler divergence alone does not establish useful prediction.

In variational inference, a tractable trial probability distribution $R$ commonly minimizes $D(R\Vert \Pi )$ against a target posterior distribution $\Pi$. Maximizing the evidence lower bound accomplishes this without evaluating the target's normalization constant. The averaging now follows $R$: placing trial probability where the posterior distribution assigns almost none is strongly penalized. The chosen direction and trial family therefore matter. ^[ref]

These comparisons do not by themselves establish causal inference. Two statistical models can reproduce the same observed probability distribution while predicting different responses to an intervention. Choosing the smaller Kullback-Leibler divergence therefore needs additional assumptions or experimental evidence before supporting a causal claim. ^[ref]

The original reference is S. Kullback and R. A. Leibler's 1951 paper ^[ref], published in The Annals of Mathematical Statistics 22(1), pages 79–86. Its title connects this comparison with sufficient statistics: the historical setting was statistical information, before today's widespread applications in prediction and variational inference.

The linear probability score pays the reported probability of whichever binary outcome occurs. With report $p$ and observed $y\in \{0,1\}$, its reward isIt looks sensible: assign more probability to what happens and receive more reward. But the expected value under a Bernoulli distribution with true success probability $q$ isThis is an affine function of the report. When $q>1/2$, the maximum over $[0,1]$ is at $p=1$; when $q<1/2$, it is at $p=0$. At $q=1/2$, every report ties. Thus the linear probability score encourages reporting certainty about the more likely outcome and is not a proper scoring rule.

For the stipulated $q=0.7$, the expected value of the reward is $0.3+0.4p$. Honest reporting $p=0.7$ earns $0.58$ in expectation, whereas $p=1$ earns $0.7$. The forecaster improves expected reward by replacing a 70% forecast with a 100% forecast. The Brier score and logarithmic scoring rule avoid this incentive, as the expected-score diagram under proper scoring rule shows.

The logarithmic scoring rule rewards the natural logarithm of the reported probability mass or probability density at the observed outcome. The title uses the reward convention. For a binary report $p$, its negative is the logarithmic loss:For $y=1$ this is $-\ln p$; for $y=0$ it is $-\ln (1-p)$. At an endpoint, use these outcome-specific forms: assigning zero probability to the outcome incurs infinite loss, while assigning it probability one gives zero. The natural logarithm therefore makes a confidently wrong binary forecast far more costly than the bounded Brier score does.

Under the stipulated Bernoulli distribution with $q=0.7$, the expected logarithmic loss isIts unique minimum is at $p=0.7$, where the expected loss is approximately $0.610864$. At $p=0.2$ it is approximately $1.193550$. More generally, the excess expected logarithmic loss over truthful reporting is the Kullback-Leibler divergence in the order true law first, reported law second:For $0<q<1$, this Kullback-Leibler divergence is nonnegative and vanishes only at $p=q$, explaining why the logarithmic scoring rule is a strictly proper scoring rule. The same Kullback-Leibler divergence identity applies to probability density functions relative to a common measure when the truthful expected logarithmic loss is finite and the comparison expected value is well defined.^[ref] For sequential forecasts, the existing prequential log score identity explains how logarithmic rewards add over observations.

A maximum-likelihood estimator is an estimator whose value maximizes the likelihood function for each sample where a finite maximizer exists. Before observation, $\hat{\theta }(X)$ is a random variable (the choice of maximizer must be a measurable function); after observation, $\hat{\theta }(x)$ is the numerical maximum-likelihood estimate. A formula must specify the allowed parameter set, including whether its endpoints belong to it.

For ten independent Bernoulli trials, write $K={\sum }_{i=1}^{10}{X}_i$. On the closed parameter interval $[0,1]$, the binomial proportion maximum-likelihood estimator is $\hat{p}=K/10$. Eight successes give the estimate $0.8$. If $K=0$, the likelihood is $(1-p{)}^{10}$ and has its unique maximum at $p=0$; if $K=10$, it is $p^{10}$ and has its unique maximum at $p=1$. On the open interval $(0,1)$ these two samples have no maximum-likelihood estimate: the supremum is one, but it is not attained. This is the same issue as likelihood supremum at an excluded boundary.

A complete continuous example uses independent and identically distributed random variables $X_i\sim N(\mu ,1)$ with unknown mean $\mu \in \mathbb{R}$ and known standard deviation one. Their likelihood function and log-likelihood areThe sum of squared deviations separates asbecause ${\sum }_i(x_i-\bar{x})=0$. The first term is fixed in $\mu$, so the unique maximum-likelihood estimator is the sample mean $\hat{\mu }=\overline{X}$. This also proves the equivalence with ordinary least squares for this model. With the constructed observations $[-1,0,2,3]$, the sample mean is one and the residual sum of squares is ten; henceAt $\mu =0$ or $2$, the log-likelihood is two below its peak, so the likelihood ratio to the fitted value is $e^{-2}\approx 0.135335$. The absolute likelihood at the peak is $(2\pi {)}^{-2}{e}^{-5}\approx 0.000170674$, a joint density value, not the probability of observing exactly those four real numbers.

The following figure compares that sample with a constructed sixteen-observation sample consisting of four repetitions of the displayed four values, modeled as sixteen independent draws. Both fits have sample mean one, butThus the observed information is four times as large for $n=16$ as for $n=4$. The distance from the peak at relative height $e^{-2}$ decreases from one to $1/2$: the width scales as $1/\sqrt{n}$. This is a comparison of two stipulated samples; merely copying recorded observations would not create new independent evidence.

Even uniqueness needs checking. If the Bernoulli distribution success parameter is written $p={\theta }^2$ with $\theta \in [-1,1]$, eight successes give two maximum-likelihood estimates, $\theta =\pm \sqrt{0.8}$, because both give the same probability distribution. The problem is a failure of identifiability. The special cases below show other ways an MLE can have bias or fail the usual asymptotic normality.

Moment matching for a beta prior turns a desired mean and variance for an unknown probability into the parameters of a beta distribution prior. Choose $0<m<1$ andSolving the beta distribution mean and variance identities givesThe strict upper bound on the variance makes $\kappa >0$, so both parameters are positive. Substitution gives mean $\mathbb{E}[P]=m$ and variance $\mathrm{Var}(P)=v$ exactly. A smaller specified variance at fixed mean produces a more concentrated prior distribution.

For example, a desired prior mean $m=0.3$ and variance $v=0.01$ give $\kappa =0.21/0.01-1=20$, hence the beta distribution prior $\mathrm{Beta}(6,14)$. With the same mean and variance $v=0.035$, the calculation instead gives $\kappa =5$ and the broader beta distribution prior $\mathrm{Beta}(1.5,3.5)$. Both appear in the concentration diagram above.

The excluded boundary cases have clear meanings. Zero variance would require $P=m$ with certainty, not a beta distribution with finite positive parameters. At the other boundary, any random variable $P\in [0,1]$ satisfies $P^2\le P$, hence variance at most $m(1-m)$; equality requires $P$ to take only the endpoint values zero and one. This is a Bernoulli distribution, not a beta distribution. Thus $v\ge m(1-m)$ cannot be matched by positive finite parameters, and $m=0$ or $m=1$ is also excluded.

After observing $n=h+t>0$ Bernoulli trials, the Beta-binomial conjugacy makes the posterior mean a weighted average of the prior mean $m$ and observed binomial proportion $h/n$:This formula gives $\kappa$ a precise interpretation as the prior's weight relative to the sample size in this posterior mean. It does not require fictitious observed counts: the beta distribution parameters can be any positive real numbers. For the eight-head, two-tail example above, $\kappa =4$ and $m=1/2$, so the posterior mean is $(4\cdot 1/2+10\cdot 0.8)/14=5/7$.

// Target: statistical-inference.bigb

Assign each edge $e$ an independent uniform distribution label $U_e$ on $[0,1]$. At parameter $p$, declare $e$ open exactly when $U_e\le p$. Each open/closed decision then has the required Bernoulli distribution, and different edge decisions are independent. The same construction applies to site percolation with labels on vertices.

Keep those labels fixed when increasing $p$. For $p\le q$, every $p$-open edge is also $q$-open, so every $p$-open graph path survives and each percolation cluster can only gain vertices. This coupling of probability distributions compares complete configurations on one probability space, rather than comparing two unrelated pictures. Any increasing event, such as an open path between specified vertices, consequently has a nondecreasing probability.

The three panels use one seeded collection of labels on a finite $17\times 17$ square lattice box. Colored edges and dots show the central root's percolation cluster; gray edges are open but disconnected from it. A percolation cluster can grow by many vertices when one newly opened edge joins it to another large percolation cluster. The panels illustrate a nested construction, not an estimate of $\theta (p)$ or a proof of the Harris-Kesten theorem.

A trigger changes which particle populations are common in the recorded sample. Let $N$ be the number of particles in an event, and stipulate an independent Bernoulli distribution for each particle's ability to produce an accepted telescope signal, with success probability $ϵ$. If any successful particle suffices, the event acceptance isThis is a deliberately simplified model. Its $ϵ$ includes the joint geometrical acceptance and response of the whole telescope; it is not an assumed universal single-counter efficiency. Actual shower directions, spatial correlations and shared electronics can violate the independence assumption.

The conditional probability formula givesFor chosen underlying fractions $\mathrm{Pr}(N=1)=0.90$, $\mathrm{Pr}(N=5)=0.10$ and $ϵ=0.15$, the two acceptances are $0.15$ and $0.5563$. Five-particle events consequently form about $29.18\%$ of recorded events. A population that is only $10\%$ of incoming events can occupy nearly $30\%$ of the photographs. This is selection bias, not evidence that the source population changed.

Under a product measure on a finite Boolean lattice, an increasing event $E$ and a decreasing event $G$ obey the displayed inequality. The complement $G^c$ is an increasing event, so Harris' inequality gives $\mathbb{P}(E\cap G^c)\ge \mathbb{P}(E)\mathbb{P}(G^c)$. Subtract from $\mathbb{P}(E)$ to obtain the result. This implication includes degenerate Bernoulli distribution parameters $0$ and $1$.