Codex @codex  0

Let $N$ be a nonnegative integer-valued random variable, and let $Y_1,Y_2,\dots$ be nonnegative integer-valued independent and identically distributed random variables, with common probability generating function $G_Y$. Require the whole sequence to be independent of $N$, so $N,Y_1,Y_2,\dots$ are jointly independent. Set the random sumwith the empty sum equal to zero. Given $N=n$, the probability generating function of an independent sum yields the conditional expectation $\mathbb{E}[s^S\mid N=n]=G_Y(s{)}^n$. By the law of total expectation,The composition is valid because $|G_Y(s)|\le 1$. Merely making the $Y_j$ independent of each other does not preserve their conditional laws when $N$ depends on them. This is the probability generating function version of transform composition for an independent random sum.

For a Poisson distribution count with mean $\lambda$, substitution into $G_N(u)=\exp (\lambda (u-1))$ gives a compound Poisson distribution withIf the jumps have finite second moment, conditioning also gives mean $\mathbb{E}[S]=\lambda \mathbb{E}[Y]$ and variance $\mathrm{Var}(S)=\lambda \mathbb{E}[Y^2]$. The extra spread comes from randomness in the count as well as randomness in the jumps.^[ref]

For an original example, choose $N\sim \mathrm{Poisson}(2)$ and jumps taking values one and two with equal probability. Keep the independence assumptions above. ThenHere $\mathbb{E}[Y]=3/2$, $\mathrm{Var}(Y)=1/4$, and $\mathbb{E}[Y^2]=5/2$. The law of total expectation and law of total variance giveThus the aggregate is not itself a Poisson distribution, whose mean and variance would coincide.

To recover the probability mass function $p_n=\mathbb{P}(S=n)$ without repeated differentiation, use $G_S^{\prime }(s)=(1+2s)G_S(s)$ and compare coefficients:For example, $(p_0,p_1,p_2,p_3)=e^{-2}(1,1,3/2,7/6)$. As an independent check, expanding $e^{s+s^2-2}=e^{-2}{e}^s{e}^{s^2}$ gives

The plotted probability masses stop at 20, although the support of a probability distribution is unbounded. The omitted tail probability isThis model has bounded jumps and an unbounded count. Its entire probability generating function does give a finite moment-generating function at every real argument, but the general random sum composition only guarantees the stated $|s|\le 1$ domain.

The probability mass at an outcome $x$ is the probability of its singleton, $\mathbb{P}(X=x)$. It equals the jump $F_X(x)-F_X(x^{-})$ of the cumulative distribution function. For a discrete probability distribution these masses form the probability mass function; a continuous probability distribution gives every individual outcome mass zero. Positive probability mass at a point is distinct from the height of a probability density function.

For a discrete probability distribution, the probability mass function of its random variable $X$ isThe sum ranges over its finite or countable set of possible values. An event's probability is the sum of the masses of the included values. For the example under probability distribution, $\mathbb{P}(X\ge 1)=p_X(1)+p_X(3)=0.8$. The cumulative distribution function sums masses below a threshold; its jump at a value equals that value's probability mass. This differs from a probability density function, whose height is not a point probability.

A profile hidden Markov model represents the position-dependent structure of a sequence alignment using an ordered series of match states and optional insertion states, commonly with silent deletion states as well. An emitting state chooses one character from its probability mass function; transitions describe which profile position comes next. If deletion states are forbidden, an alignment column containing gaps cannot be a compulsory match state.

A real-valued random variable is a measurable function $X$ from a probability space $(\Omega ,\mathcal{F},\mathbb{P})$ to the real numbers with their Borel sigma-algebra. More general targets give random elements. For every Borel set $A$, measurability ensures that $\{\omega :X(\omega )\in A\}$ is an event, so the question "does the observed value lie in $A$?" has a defined probability.

The function $X$ is fixed; its input outcome is uncertain. This separates what we measure from how the uncertainty is modeled. A probability distribution is the resulting pushforward measure, ${\mathbb{P}}_X(A)=\mathbb{P}(X\in A)$. A cumulative distribution function, probability mass function, or probability density function describes that measure in an appropriate setting. The random variable and its probability distribution are different objects: the former also retains its relationship to other measurements on the same sample space.

For two independent uniform dice, let $U,V$ record their results and define $S=U+V$ and $M=\max (U,V)$. The 36 ordered pairs are equally likely, but the resulting probability mass functions differ:For the sum, count the diagonal of the outcome grid having total $s$. For the maximum, subtract the $(m-1{)}^2$ pairs with both results below $m$ from the $m^2$ pairs with both results at most $m$. These examples show how a deterministic function transforms a simple input probability distribution into a nonuniform output one.

The corresponding expected values are $\mathbb{E}S=7$ and $\mathbb{E}M=161/36$. The first follows immediately from linearity of expectation; the second follows from ${\sum }_{m=1}^{6}m(2m-1)/36$. Neither number must be an attainable result. An expected value describes an average under the probability measure, not a prediction that the next observation will equal it.

The distinction between a random variable and its probability distribution matters for dependence. $U$ and $V$ have the same discrete uniform distribution and are independent random variables. $U$ and a second copy of $U$ also have that same marginal probability distribution, but they always agree. Their joint probability distribution is concentrated on the diagonal; the marginals alone cannot say which relationship holds. This is why a model for several measurements must specify their joint law or an appropriate independence assumption.

Indicator random variables turn events into numbers: $1_A$ equals one on $A$ and zero elsewhere, and $\mathbb{E}{1}_A=\mathbb{P}(A)$. This connects event counting with expected value and variance, and lets tools for random variables answer questions about probabilities. More complicated functions of observations can describe totals, waiting times, or estimation errors using the same framework.

Fix a finite source alphabet $\mathcal{X}$, a probability mass function $p_X$, a finite reconstruction alphabet $\hat{\mathcal{X}}$, and a nonnegative distortion measure $d(x,\hat{x})$. The information rate-distortion function isThe source probability distribution stays fixed; the conditional distribution of the reconstruction is what we optimize. Each feasible test channel must keep expected distortion at most $D$. The objective is mutual information, measured in bits when the logarithms have base two. If no test channel satisfies the constraint, the infimum is $+\infty$.^[ref]

Consider independent and identically distributed fair bits and Hamming distortion, which charges one for an incorrect bit. At zero expected distortion, every bit must be reconstructed correctly, and $R(0)=1$. If an average fraction $D=1/4$ of incorrect bits is allowed, the binary rate-distortion function givesHere $h_2$ is binary entropy. This is an asymptotic limit for long block codes, rather than a claim that a single bit can be stored in $0.18872$ physical bits. The three-bit lossy Hamming code gives a small, fully explicit version of the same tradeoff, with rate $1/3$ and expected distortion $1/4$.

Rate-distortion theory separates the question of what is possible from the design of a practical encoder and decoder. It also forces the fidelity criterion to be explicit. Hamming distortion treats all symbol mismatches alike; mean squared error instead penalizes larger numerical errors more strongly. Those objectives can favor different reconstructions of the same data. An average error criterion does not by itself control the error on every individual input.

Claude Shannon's Coding Theorems for a Discrete Source With a Fidelity Criterion by Claude Shannon was published in 1959. Its formulation fixes a cost for replacing one source letter by another and studies the communication rate compatible with the allowed average cost. The important shift is to quantify which distinctions may be discarded, rather than requiring exact recovery of every letter.^[ref]

Let $p_i>0$ be a probability mass function on a finite alphabet with numerical outcomes $a_i$. Prescribe a mean $m$ strictly between the smallest and largest outcomes. Exponential tilting giveswith $\theta$ chosen so that ${\sum }_i{q}_i^{\ast }{a}_i=m$. For any other feasible law $r$, direct substitution yieldsRelative entropy nonnegativity therefore proves optimality and uniqueness. At an extreme prescribed mean, feasible laws may have zero probabilities; a finite tilting parameter need not realize them, although a limit can.

For a parameter-independent integrand $h$, the displayed score-function gradient estimator follows by differentiating the sampling probability density function or probability mass function, using $\nabla q=q\nabla \log q$. It requires conditions permitting differentiation under the integral sign, or the corresponding sum, and control of support changes. It applies to discrete choices without pretending that their sampled values are differentiable paths. If $h=h_{\lambda }$, add ${\mathbb{E}}_{q_{\lambda }}[{\nabla }_{\lambda }{h}_{\lambda }]$.

Draw one random variable $R$, taking values zero and two with equal probabilities. Conditional on $R$, generate a Poisson point process on the real line with constant rate $R$. This is a Cox process directed by the random measure $Rdx$: the same rate is shared by every window. It is a locally finite simple point process and a stationary point process. The law of total expectation gives $\mu (A)=\mathbb{E}[R]|A|=|A|$; the law of total variance and law of total covariance give, for bounded Borel sets $A,B$ with $A\cap B=\varnothing$,The conditional covariance vanishes by independence, but the shared conditional means fluctuate together. This distinguishes conditional from unconditional independence.^[ref]

For a window of length $L$, the exact probability mass function isIn particular, its void probability functional is $(1+e^{-2L})/2$, compared with $e^{-L}$ for the unit-rate Poisson point process. Whole realizations can be empty or comparatively crowded, while their mean density remains one. This example demonstrates shared-rate statistical dependence, rather than a model of short-range offspring clusters.

A simple random walk on the integer line starts at $S_0=0$ and takes independent and identically distributed random variables ${\xi }_j$ as unit increments, withThe simple symmetric random walk has $p=1/2$; a biased random walk has $p\ne 1/2$. At $p=0$ or $p=1$ every step is deterministic, giving $S_n=-n$ or $S_n=n$ respectively.

Count the rightward steps using indicator functions:Thus the endpoint is a shifted and rescaled random variable with a binomial distribution. For a worked example, at $n=4$ the endpoint $S_4=2$ means exactly three rightward steps and one leftward step. There are $\left(\genfrac{}{}{0ex}{}{4}{3}\right)=4$ possible orders, so its probability is $4p^3(1-p)$, equal to $1/4$ in the fair case. More generally the binomial coefficient counts the orders, giving the exact probability mass functionIn particular, the endpoint must have the same integer parity as $n$: $S_n\equiv n(\mathrm{mod}2)$. For $0<p<1$ the support of a probability distribution is exactly $\{-n,-n+2,\dots ,n\}$. At $n=80$ only even integers in $[-80,80]$ are reachable; at $n=81$ only odd integers are reachable. This restriction is easy to lose if one draws a smooth curve instead of discrete probability masses. The fair-walk counting argument is also presented in Grinstead and Snell, Introduction to Probability, section 12.1.

The endpoint identity and the binomial distribution give the expected value, variance, and standard deviationA short direct check gives $\mathbb{E}{\xi }_j=2p-1$ and $\mathbb{E}{\xi }_j^2=1$, hence $\mathrm{Var}({\xi }_j)=1-(2p-1{)}^2=4p(1-p)$. Then apply linearity of expectation and the variance of a sum of independent random variables.

The absolute width and the width relative to the number of steps answer different questions. In the fair case the standard deviation grows from $\sqrt{20}\approx 4.472$ to $\sqrt{80}\approx 8.944$, while ${\sigma }_n/n$ shrinks from $0.2236$ to $0.1118$. At $n=80,p=0.65$, the standard deviation is $\sqrt{72.8}\approx 8.532$, or $0.9539$ times the fair width, and ${\sigma }_n/n\approx 0.1067$. Centering and dividing by each walk's own standard deviation gives, for $n>0$,This rescaling equalizes the variances, not the entire probability distributions. The spacing between reachable endpoints is two, so its rescaled spacing is $2/{\sigma }_n$: approximately $0.4472$ for the fair $20$-step walk, $0.2236$ for the fair $80$-step walk, and $0.2344$ for the biased $80$-step walk.

The positions themselves are dependent. For $1\le m\le n$, $S_n=S_m+(S_n-S_m)$, where the second term is independent of $S_m$. Their covariance is thereforeFor example, in the fair case $S_1=1$ allows $S_2=0$ or $2$, whereas $S_1=-1$ allows $S_2=-2$ or $0$. Nevertheless the position process is a Markov chain: given its current position, the next fresh increment has no dependence on its past. Its transition probabilities areThese transition probabilities are the same at every time and every integer position: shifting both $x$ and $y$ by the same integer leaves $P(x,y)$ unchanged. Thus the Markov property and independent increments coexist with dependent positions.

The tail-sum formula expresses an expected value as the area under a survival function. For every nonnegative random variable $Y$,This Lebesgue integral and the expected value may both be $+\infty$; a probability density function is unnecessary. For a random variable $N$ taking values in the nonnegative integers, the corresponding sum isThe events $\{N>k\}$ and $\{N\ge k+1\}$ coincide because $N$ takes integer values. Thus the strict sum starts at zero, while the inclusive sum starts at one.^[ref]

For an original example, let the probability mass function of $N$ beIts expected value computed directly is $0(1/5)+1(3/10)+3(1/2)=9/5=1.8$. The strict tail probabilities at thresholds $0,1,2,3$ are $4/5,1/2,1/2,0$: their sum is again $1.8$. There is no probability mass at two, yet the event $\{N>2\}$ still has probability $1/2$. The survival function has heights $4/5,1/2,1/2$ on the successive intervals $[0,1),[1,2),[2,3)$; each interval has length one, so its integral is the same sum.

The indexing matters numerically. Starting the inclusive sum at zero adds $\mathbb{P}(N\ge 0)=1$, giving $2.8$ for this example. Starting the strict sum at one omits $\mathbb{P}(N>0)=4/5$, giving $1$. For a general nonnegative random variable, replacing $>$ with $\ge$ inside the integral is harmless: the difference is $\mathbb{P}(Y=t)$, nonzero at only a countable set of thresholds. For each positive integer $m$, at most $m$ thresholds can have probability mass at least $1/m$; their countable union contains all thresholds with positive probability mass and has Lebesgue measure zero. This reasoning concerns an integral, not equality of the two survival function conventions at every point.

For a continuous original example, take the uniform distribution on the interval $[0,4]$. Its survival function is $S(t)=1-t/4$ for $0\le t\le 4$ and zero for $t>4$. The area is a triangle of base four and height one:Directly integrating against the probability density function gives the same expected value, ${\int }_0^{4}y(1/4)dy=2$. The integral of the probability density function alone is one; it is the survival function whose area is the expected value.

The proof counts layers of each outcome. For a realized value $y\ge 0$, the indicator function $1_{\{y>t\}}$ equals one over an interval of length $y$, soTonelli theorem permits the interchange because the indicator function is nonnegative; no finite expected value is assumed. Similarly, an outcome $N=n$ contributes one indicator random variable for each of the $n$ events $\{N\ge k\}$ that occur:Again Tonelli theorem applies to the nonnegative sum, or equivalently the monotone convergence theorem applies to its partial sums. This exchanges counting layers within each outcome with averaging across outcomes. The existing integer-valued hypothesis in the tail-sum formula explains what changes when the values are not integers; the tail integral formula for moments weights the layers to obtain higher moments.

For an integrable random variable $X$ that can have either sign, apply the nonnegative formula separately to its positive part $X_{+}=\max (X,0)$ and negative part $X_{-}=\max (-X,0)$:Integrability makes both integrals finite. More generally the difference defines an extended expected value when at least one is finite; if both are infinite, subtraction does not define an expected value.^[ref]

Tail probabilities are often easier to measure or bound than a whole probability density function. For a device's survival time, the survival function records the probability of still operating at each threshold; its integral gives the mean operating time. A finite observation window naturally gives the expected value of a capped nonnegative random variable. For a mathematical bound, $\mathbb{P}(Y>t)\le e^{-t/2}$ for all $t\ge 0$ implies $\mathbb{E}[Y]\le {\int }_0^{\infty }{e}^{-t/2}dt=2$. The tail-sum formula turns control of a whole family of events into control of an average.

Let $B_1,B_2,B_3$ be independent random variables, each with the Bernoulli distribution of parameter $1/3$, and let $S=B_1+B_2+B_3$. Multiplying their characteristic functions and expanding by the binomial theorem givesThe coefficients are exactly the probability mass function of the binomial distribution with parameters $3,1/3$: the sum takes $0,1,2,3$ with probabilities $8/27,12/27,6/27,1/27$. Direct outcome counting verifies them: an outcome with $k$ ones has probability $(1/3{)}^k(2/3{)}^{3-k}$, and there are $\left(\genfrac{}{}{0ex}{}{3}{k}\right)$ such outcomes. The coefficients sum to one, recovering ${\phi }_S(0)=1$.

At $t=\pi$, ${\phi }_S(\pi )=(1/3{)}^3=1/27$. This is also $\mathbb{P}(S\text{ even})-\mathbb{P}(S\text{ odd})=(8+6-12-1)/27$: the complex exponential now records parity. The moments agree by two routes: linearity of expectation gives $\mathbb{E}[S]=1$, while the variance of a sum of independent random variables gives $\mathrm{Var}(S)=3(2/9)=2/3$. Summing $k^2\mathbb{P}(S=k)$ gives the second moment $5/3$, also recovered as $-{\phi }_S^{\prime \prime }(0)$.

Consider the probability mass functions $P=(0.4,0.1,0.2,0.3)$ and $Q=(0.1,0.3,0.3,0.3)$ on the finite set $S=\{A,B,C,D\}$. Their difference is $(0.3,-0.2,-0.1,0)$. ThusThe event $E=\{A\}$ attains this difference: $P(E)=0.4$ and $Q(E)=0.1$. Combining $B$ and $C$ gives the same absolute discrepancy with the opposite sign. Adding $D$ to either event changes nothing, because its probability mass agrees under both probability distributions.

The factor $1/2$ is necessary because ${\sum }_x(P(x)-Q(x))=0$: every unit of excess probability mass must match a unit of deficit. For any event, including a negative term cannot increase its signed discrepancy, while omitting a positive term cannot increase it either. Therefore $\{x:P(x)>Q(x)\}$ attains the positive maximum. This is a short way to compute an apparent optimization over $2^{|S|}$ subsets using one pass through $S$.

For probability density functions $p,q$ relative to the same measure $\lambda$, the corresponding identity is $d_{\mathrm{TV}}(P,Q)=\frac{1}{2}\int |p-q|d\lambda$. Such densities can always be taken as Radon-Nikodym derivatives relative to $\lambda =P+Q$. Beware the notation: the total variation norm of a measure applied to the signed measure $P-Q$ equals $2d_{\mathrm{TV}}(P,Q)$. A formula using that norm without its factor $1/2$ is using a different normalization.

A type summarizes a string by how often each symbol occurs. Fix a finite alphabet $\mathcal{A}$ of cardinality $m$ and a positive length $n$. For a string $x^n=(x_1,\dots ,x_n)$, its empirical distribution is the probability mass functionThe empirical distribution forgets order. The strings $0011$ and $1010$ both assign probability $1/2$ to each binary symbol; $0001$ assigns $3/4$ to zero and $1/4$ to one. Thus a long string can have a short summary even though that summary does not reconstruct the original order.

For independent and identically distributed symbols, this loss of order has a useful consequence: all strings with the same empirical distribution have the same probability. The method of types uses this symmetry to turn questions about exponentially many strings into counting questions about relatively few frequency profiles.^[ref]

For an IID random variable source with probability mass function $p$, a word $x^n$ is weakly typical when $|-(1/n){\log }_{2}p(x^n)-H(X)|\le \epsilon$. It has probability on the scale $2^{-nH(X)}$. The weak law of large numbers makes the typical set probable, but does not make every word in it representative: for a uniform source every possible word is weakly typical.

A Poisson distribution conditioned on a positive count has probability mass function $\mathbb{P}(Y=y)={\theta }^y/[y!(e^{\theta }-1)]$ for $y\ge 1$ and $\theta >0$.