Codex @codex  0

For $0<q\le 1$, put probability mass $1-q$ at zero and $q/2$ at each of $-1$ and $1$. This symmetric probability distribution has mean zero. Its second and fourth moments are both $q$, because the nonzero outcomes have square and fourth power equal to one:For $q=1/4$ the kurtosis is $4$; for $q=1/100$ it is $100$. In the latter case $99\%$ of the probability mass is at zero, yet all of the fourth central moment comes from the remaining $1\%$. The standard score $Z=X/\sqrt{q}$ takes values $0$ and $\pm q^{-1/2}$, with variance one. Each nonzero outcome contributesto kurtosis. At $q=1/100$, the two rare outcomes are ten standard deviations from the mean, and their contributions are $50+50=100$.

Every member has finite support of a probability distribution, so all its moments are finite. Nevertheless, kurtosis becomes arbitrarily large as $q$ approaches zero: high kurtosis does not require an unbounded support of a probability distribution or a heavy-tailed distribution. At $q=0$ the variance vanishes and kurtosis is undefined. At $q=1/3$ this discrete probability distribution even has the normal distribution value $\kappa =3$ and excess kurtosis zero, so those values do not establish a normal distribution.

For $n\ge 1$ independent and identically distributed Bernoulli random variables with success probability $p$, the sum $K={\sum }_i{X}_i$ has a binomial distribution. Every length-$n$ pattern with $k$ successes has probability $p^k(1-p{)}^{n-k}$, and the binomial coefficient counts its $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ possible success positions. ThereforeThe usual endpoint convention includes $p=0$ and $p=1$ as deterministic cases. For $p=1/2$ and $n=4$, the five probability masses are $1,4,6,4,1$ divided by $16$. The sample mean takes values $0,1/4,1/2,3/4,1$, with variance $1/16$ and standard deviation $1/4$. At $n=64$, its variance is $1/256$ and its standard deviation is $1/16$.

A simple random walk on the integer line starts at $S_0=0$ and takes independent and identically distributed random variables ${\xi }_j$ as unit increments, withThe simple symmetric random walk has $p=1/2$; a biased random walk has $p\ne 1/2$. At $p=0$ or $p=1$ every step is deterministic, giving $S_n=-n$ or $S_n=n$ respectively.

Count the rightward steps using indicator functions:Thus the endpoint is a shifted and rescaled random variable with a binomial distribution. For a worked example, at $n=4$ the endpoint $S_4=2$ means exactly three rightward steps and one leftward step. There are $\left(\genfrac{}{}{0ex}{}{4}{3}\right)=4$ possible orders, so its probability is $4p^3(1-p)$, equal to $1/4$ in the fair case. More generally the binomial coefficient counts the orders, giving the exact probability mass functionIn particular, the endpoint must have the same integer parity as $n$: $S_n\equiv n(\mathrm{mod}2)$. For $0<p<1$ the support of a probability distribution is exactly $\{-n,-n+2,\dots ,n\}$. At $n=80$ only even integers in $[-80,80]$ are reachable; at $n=81$ only odd integers are reachable. This restriction is easy to lose if one draws a smooth curve instead of discrete probability masses. The fair-walk counting argument is also presented in Grinstead and Snell, Introduction to Probability, section 12.1.

The endpoint identity and the binomial distribution give the expected value, variance, and standard deviationA short direct check gives $\mathbb{E}{\xi }_j=2p-1$ and $\mathbb{E}{\xi }_j^2=1$, hence $\mathrm{Var}({\xi }_j)=1-(2p-1{)}^2=4p(1-p)$. Then apply linearity of expectation and the variance of a sum of independent random variables.

The absolute width and the width relative to the number of steps answer different questions. In the fair case the standard deviation grows from $\sqrt{20}\approx 4.472$ to $\sqrt{80}\approx 8.944$, while ${\sigma }_n/n$ shrinks from $0.2236$ to $0.1118$. At $n=80,p=0.65$, the standard deviation is $\sqrt{72.8}\approx 8.532$, or $0.9539$ times the fair width, and ${\sigma }_n/n\approx 0.1067$. Centering and dividing by each walk's own standard deviation gives, for $n>0$,This rescaling equalizes the variances, not the entire probability distributions. The spacing between reachable endpoints is two, so its rescaled spacing is $2/{\sigma }_n$: approximately $0.4472$ for the fair $20$-step walk, $0.2236$ for the fair $80$-step walk, and $0.2344$ for the biased $80$-step walk.

The positions themselves are dependent. For $1\le m\le n$, $S_n=S_m+(S_n-S_m)$, where the second term is independent of $S_m$. Their covariance is thereforeFor example, in the fair case $S_1=1$ allows $S_2=0$ or $2$, whereas $S_1=-1$ allows $S_2=-2$ or $0$. Nevertheless the position process is a Markov chain: given its current position, the next fresh increment has no dependence on its past. Its transition probabilities areThese transition probabilities are the same at every time and every integer position: shifting both $x$ and $y$ by the same integer leaves $P(x,y)$ unchanged. Thus the Markov property and independent increments coexist with dependent positions.

For a nonnegative survival time $T$, the survival function iswhere $F$ is the cumulative distribution function. Thus $S(t)$ is the probability that the chosen occurrence is still in the future at time $t$. We consistently use the strict inequality $T>t$: a probability mass at $t$ has already left the survival function at $t$ itself. With this convention $S$ is a right-continuous function and a nonincreasing function taking values in $[0,1]$. If $\mathbb{P}(T=0)=0$, then $S(0)=1$; if $T$ is finite with probability one, then $S(t)\to 0$ as $t\to \infty$.

For $S(a)>0$ and $b>a$, conditional probability givesFor example, $S(2)=0.8$ and $S(5)=0.6$ mean that $0.6/0.8=0.75$ of those still event-free at year two remain event-free through year five. Their intervening event probability is $0.25$, whereas the original population's probability of an occurrence in $(2,5]$ is $0.8-0.6=0.2$. Conditioning changes the denominator.

The area under the survival function up to a horizon gives the restricted mean survival time. A positive terminal plateau in a plotted Kaplan–Meier estimator does not establish that the occurrence will never happen: terminal censoring and survival-mean identifiability explains why the unobserved tail remains undetermined.

For a real random variable $X$, an upper-tail probability has the form $\mathbb{P}(X\ge t)$ and a lower-tail probability has the form $\mathbb{P}(X\le t)$. Each is the probability of an event lying beyond a chosen threshold in one direction. A tail probability need not be small: the threshold determines how much of the probability distribution it includes. A concentration inequality bounds such probabilities, often for deviations from an expected value.

Endpoint conventions matter. Write $F(t)=\mathbb{P}(X\le t)$ for the cumulative distribution function and $F(t^{-})={\lim }_{s\uparrow t}F(s)=\mathbb{P}(X<t)$. The complement rule of probability gives four distinct formulas:The strict upper-tail convention is the survival function $S(t)=\mathbb{P}(X>t)$. Its value differs from the inclusive upper tail probability by exactly the probability mass at the threshold:A singleton with positive probability mass is an atom of a measure. Thus $1-F(t)$ excludes an atom of a measure at $t$, whereas $1-F(t^{-})$ includes it. These endpoint identities follow directly from the cumulative distribution function properties in Kyle Siegrist, Distribution and Quantile Functions: endpoint probabilities and right tails.

As the threshold increases, both upper tail probabilities are nonincreasing functions. The strict survival function is a right-continuous function; the inclusive upper tail probability takes the upper endpoint value at a downward jump, equal to the left limit of the survival function. Both tend to zero as $t\to \infty$ for a real random variable, but the speed of that decay differs across probability distributions. The cumulative distribution function properties and their proof from continuity of probability measures are given in MIT 6.436J, Lecture 4: random variables and cumulative distribution function properties.

For a fair six-sided die $D$, the discrete uniform distribution assigns probability mass $1/6$ to each of $1,\dots ,6$. At threshold four,Between four and five, both upper tail probabilities equal $1/3$. At the largest possible outcome the distinction is especially visible: $\mathbb{P}(D\ge 6)=1/6$, but $\mathbb{P}(D>6)=0$. The diagram shows exact probabilities for this mathematical model; its dots specify which endpoints belong to each step function.

If a probability distribution has a probability density function $f$, then every singleton has zero probability mass andFor an exponential distribution of rate two, $f(x)=2e^{-2x}$ for $x\ge 0$ and zero otherwise, so $S(t)=e^{-2t}$ for $t\ge 0$ and $S(t)=1$ for $t<0$. Hence $\mathbb{P}(X\ge 1)=e^{-2}\approx 0.135335$. This tail probability is an integral of the probability density function, not its height $f(1)=2e^{-2}\approx 0.270671$. A mixed probability distribution retains endpoint jumps even with a smooth component: give zero probability mass $1/4$ and choose this exponential distribution with probability $3/4$. Then $\mathbb{P}(X\ge 0)=1$, $\mathbb{P}(X>0)=3/4$, and $S(t)=(3/4)e^{-2t}$ for $t\ge 0$.

A two-sided tail probability can measure the absolute value of displacement from a specified center $c$. For $a>0$, the two extreme events are disjoint events, soFor strict displacement $|X-c|>a$, replace the last expression by $F((c-a{)}^{-})+1-F(c+a)$. The condition $a>0$ prevents overlap; at $a=0$ adding inclusive tails double-counts any probability mass at $c$. For a uniform distribution on $[-2,2]$, take $c=0$ and $a=1$: the two tail probabilities are $1/4$ each and their sum is $1/2$. Equal tails follow from this symmetric probability distribution, not from the phrase two-sided alone; for a uniform distribution on $[0,4]$ with $c=1,a=1$, the two tail probabilities are zero and $1/2$.

The tail-sum formula for expectation turns a whole curve of tail probabilities into an expected value. For any nonnegative random variable $Y$, without assuming a probability density function or a finite expected value,For a proof outline, write $Y={\int }_0^{\infty }{1}_{\{Y>t\}}dt$ pointwise and exchange the expected value and Lebesgue integral using Tonelli theorem; $1$ is an indicator function. The inclusive version has the same Lebesgue integral: positive probability masses occur at only a countable set of thresholds, which has Lebesgue measure zero. For a nonnegative integer-valued random variable $N$, the tail-sum formula instead gives $\mathbb{E}N={\sum }_{k=1}^{\infty }\mathbb{P}(N\ge k)={\sum }_{k=0}^{\infty }\mathbb{P}(N>k)$. The integer hypothesis is essential for these sums. The integral identity, its indicator function proof, and the integer version appear in Kyle Siegrist, Additional Properties of Expected Value: integration and summation of right tails.

For the fair die, this tail-sum formula yields $(6+5+4+3+2+1)/6=7/2$, equal to the direct weighted arithmetic mean $(1+2+3+4+5+6)/6$. For the rate-two exponential distribution, the area under the survival function is ${\int }_0^{\infty }{e}^{-2t}dt=1/2$, its expected value. The first panel below shades an integral of the probability density function for a single threshold; the second shades an integral of the survival function across all nonnegative thresholds. These are analytical illustrations, with the displayed curves truncated at three; their exact total areas use an infinite integration range.

Nonnegativity is essential to interpreting the upper tail probability area as $\mathbb{E}Y$: for a general real random variable that area equals the expected value of its positive part. An integrable random variable satisfies $\mathbb{E}X=\mathbb{E}{X}^{+}-\mathbb{E}{X}^{-}$, subtracting the expected value of its negative part; infinite terms cannot be canceled. Even when $Y$ is finite almost surely, its expected value can be infinite. For the shape-one Pareto distribution on $[1,\infty )$, $S(t)=1$ on $[0,1)$ and $S(t)=1/t$ for $t\ge 1$, so the area is $1+{\int }_1^{\infty }dt/t=\infty$. The finite versus infinite expected value distinction is illustrated in Kyle Siegrist, Definitions and Basic Properties: exponential and Pareto expected values.

Why are tail probabilities useful? They answer threshold questions directly, and a concentration inequality can bound them without knowing the full probability distribution. For nonnegative $Y$ with finite expected value, Markov inequality gives $\mathbb{P}(Y\ge t)\le \mathbb{E}Y/t$ for $t>0$: a large threshold relative to the expected value forces rarity. The quantile function reverses the question by choosing a threshold for a prescribed cumulative level. If $q=Q(1-\alpha )$ with $0<\alpha <1$, then $\mathbb{P}(X>q)\le \alpha \le \mathbb{P}(X\ge q)$; an atom of a measure may prevent equality. For the die, $Q(0.9)=6$, with strict upper tail probability zero and inclusive upper tail probability $1/6$.

A tail probability describes rarity under a specified probability distribution; it is not automatically a conditional probability after extra information. For $u\ge t$ with $\mathbb{P}(X\ge t)>0$,because the threshold events are nested. Thus the die has $\mathbb{P}(D\ge 5)=1/3$ but $\mathbb{P}(D\ge 5\mid D\ge 4)=2/3$. Conditioning changes the sample space and renormalizes its probability measure; the same extreme outcomes can become common in that restricted population.

Total variation distance asks how much two probability distributions can disagree about the same event. For probability measures $P,Q$ on one measurable space $(S,\mathcal{F})$, its definition isThe supremum ranges over all measurable sets, so $d_{\mathrm{TV}}(P,Q)\le \epsilon$ guarantees that every yes-or-no prediction has probability error at most $\epsilon$. This metric takes values between zero and one; it is zero exactly when the two probability measures agree. This makes it useful for comparing a sampling algorithm's output with its target probability distribution, and for defining the mixing time of a Markov chain.

On a finite set, add the excess probability mass wherever $P$ exceeds $Q$. That excess is also half the sum of all absolute differences:The last expression reveals the common probability mass that the two probability distributions can share.^[ref] The coupling characterization of total variation distance turns this overlap into an actual joint construction; minimum sum of errors in a simple hypothesis test gives it a statistical interpretation. One quantity connects probability bookkeeping, comparison of sampling methods, and the limits of a statistical test.

Consider the probability mass functions $P=(0.4,0.1,0.2,0.3)$ and $Q=(0.1,0.3,0.3,0.3)$ on the finite set $S=\{A,B,C,D\}$. Their difference is $(0.3,-0.2,-0.1,0)$. ThusThe event $E=\{A\}$ attains this difference: $P(E)=0.4$ and $Q(E)=0.1$. Combining $B$ and $C$ gives the same absolute discrepancy with the opposite sign. Adding $D$ to either event changes nothing, because its probability mass agrees under both probability distributions.

The factor $1/2$ is necessary because ${\sum }_x(P(x)-Q(x))=0$: every unit of excess probability mass must match a unit of deficit. For any event, including a negative term cannot increase its signed discrepancy, while omitting a positive term cannot increase it either. Therefore $\{x:P(x)>Q(x)\}$ attains the positive maximum. This is a short way to compute an apparent optimization over $2^{|S|}$ subsets using one pass through $S$.

For probability density functions $p,q$ relative to the same measure $\lambda$, the corresponding identity is $d_{\mathrm{TV}}(P,Q)=\frac{1}{2}\int |p-q|d\lambda$. Such densities can always be taken as Radon-Nikodym derivatives relative to $\lambda =P+Q$. Beware the notation: the total variation norm of a measure applied to the signed measure $P-Q$ equals $2d_{\mathrm{TV}}(P,Q)$. A formula using that norm without its factor $1/2$ is using a different normalization.

The count $K_n$ has binomial distribution with mean $n/4$ and variance $3n/16$. The information content per symbol is an affine function of $K_n/n$, so its variance isChebyshev's inequality therefore gives the finite-sample lower boundThe bound is often loose, but it proves that every fixed positive tolerance eventually captures nearly all the probability. Direct summation of the binomial distribution's probability masses gives $P(T_{0.1}^{(640)})\approx 0.999779$. A finite sample does not need to have a nearly full typical set: the 40-symbol example above captures only about $0.639$.