Bayes' theorem ($\mathbb{P}(B\mid A)=\mathbb{P}(A\mid B)\mathbb{P}(B)/\mathbb{P}(A)$)

For a finite measurable partition $(B_j)$ with positive probabilities, and an event $A$ of positive probability,Its beauty is that a model predicting observations from a hidden state can also describe uncertainty about the state after an observation. Each numerator is a joint probability; the denominator, supplied by the law of total probability, makes the answers sum to one. This is a direct consequence of conditional probability, rather than an extra axiom.

Select the coin from law of total probability once, then toss that same coin repeatedly, with conditionally independent tosses given its identity. Let $B$ denote the $3/4$-head coin, initially of prior probability $1/5$. After one head $H$, its posterior probability isThe likelihood of a head is larger under $B$, but $B$ remains less likely than the other coin because its prior probability was smaller. A large likelihood for an observation does not itself imply a large posterior probability for the hypothesis.

After an ordered sequence containing $h$ heads and $t$ tails, its likelihood ratio between the two coins is $(3/2{)}^h(1/2{)}^t$. Bayes' theorem therefore gives posterior oddsEach head multiplies the odds by $3/2$ and each tail by $1/2$. Three heads give posterior probability $27/59$, still below one half; four give $81/145$, above one half. The finite-model update and the distinction between prior probability, likelihood and posterior probability are set out in MIT 18.05, Reading 11: Bayesian Updating with Discrete Priors.

The hidden selection also creates dependence. Each toss has marginal head probability $11/20$, but two heads have joint probability $(4/5)(1/2{)}^2+(1/5)(3/4{)}^2=5/16$, greater than $(11/20{)}^2=121/400$. Their indicator covariance is $1/100$. Seeing a head makes the head-favoring coin more likely, so it also raises the chance of a future head. This is covariance induced by a shared latent variable, despite conditional independence of the tosses given the coin.

Historically, Thomas Bayes's essay on an unknown success probability appeared in 1763 after his death, communicated by Richard Price. Its opening problem asks how observed successes and failures constrain that unknown probability; Price's introductory letter explains the connection to reasoning from experience. That is a more substantial inference problem than simply rearranging today's elementary identity. ^[ref]

Suppose exactly one of $N$ equally likely candidates is the source of an observation. A test is always positive for that source and is positive with probability $p$ for any nonsource candidate. For one randomly selected candidate, Bayes' theorem gives $P(G\mid +)=(1/N)/[(1/N)+p(1-1/N)]$, proving the displayed result. Even a small false positive rate can leave a substantial chance of a different source when there are many alternatives. The model concerns one selected candidate with the stated prior; searching many candidates and selecting a match requires accounting for that selection procedure.

An urn has $N$ balls, including $K$ of one colour and $N-K$ of the other, with $0<K<N$. After a uniform first draw, an independent decision returns that ball with probability $r$; the second draw is uniform from the remaining contents. Let $B_j$ mean the $j$th draw has the first colour and $R$ mean replacement. The two relevant joint probabilities areApplying Bayes' theorem and cancelling the common factors gives the displayed conditional probability. It is independent of $K$ and satisfiesTwo different observed colours therefore decrease the posterior probability of replacement: removing the first colour makes the second more likely. The replacement decision must be independent of the first draw for this formula; its marginal probability alone does not specify the model.

Let $N$ have the positive-integer geometric distribution with success parameter $p\in (0,1)$, and conditionally toss $N$ independent coins with head probability $\theta \in (0,1)$. Exactly one head has conditional probability $n\theta (1-\theta {)}^{n-1}$. Combining this with the prior by Bayes' theorem gives the displayed posterior with $r=(1-p)(1-\theta )$; its normalization uses ${\sum }_{n\ge 1}n{r}^{n-1}=(1-r{)}^{-2}$. For $p=\theta =1/2$, the posterior is $9n/4^{n+1}$. It is also a shifted negative binomial distribution, with $N-1$ equal in law to the number of failures before two successes of probability $1-r$.

If every model's likelihood is multiplied by the same positive factor independent of its parameter or model index, Bayes' theorem cancels that factor from the normalized posterior. Fixed-count Bernoulli sampling and sampling stopped at a specified positive head count give such proportional likelihoods for the same total heads and tails. This conclusion uses their actual sampling likelihoods; it is not a claim that every selection or stopping rule can be ignored.

If a hidden coin is selected once with prior probability $\pi$ of coin $B$, repeated tosses are independent conditional on that coin. Bayes' theorem weights the prior by each all-heads likelihood. They need not be independent without conditioning on the chosen coin.