= Normal mean confidence interval and hypothesis test duality
{title2=$z=(\bar x-\mu_0)/(\sigma/\sqrt n),\quad p=2[1-\Phi(|z|)]$}
For the same <normal distribution> model, a <two-sided test> compares the <null hypothesis> $H_0:\mu=\mu_0$ with the <alternative hypothesis> $H_1:\mu\ne\mu_0$. Under the <null hypothesis>, the standardized <test statistic> has <standard normal distribution>. Write $\Phi$ for the <standard normal distribution function>. With $\bar x=1.2$, $\sigma=2$, $n=16$ and $\mu_0=0$, the observed <test statistic> is $z=1.2/0.5=2.4$. Its <p-value> is
$$
p=\mathbb P_{\mu_0}(|Z|\geq2.4)
=2[1-\Phi(2.4)]
=0.01639507185\ldots.
$$
Thus the <p-value> is a tail <probability> computed under the <null hypothesis>, including departures in both directions. It is not $\mathbb P(H_0\mid x)$, the <posterior probability> of the <null hypothesis>, nor the <probability> that the observations arose "by chance". The <normal distribution> <test statistic> and its calibration appear in https://www.itl.nist.gov/div898/handbook/prc/section2/prc22.htm[NIST's teaching reference for testing a process mean]; the tail interpretation is stated in https://online.stat.psu.edu/stat200/Lesson05[Penn State's hypothesis-testing notes].
Choose the <significance level> $\alpha=0.05$ before observing the sample. The <rejection region> is $|Z|>z_{0.975}$, whose <probability> under $H_0$ is exactly 0.05. Here $2.4>1.9599639845\ldots$, so reject $H_0$. This agrees with zero lying outside $[0.22,2.18]$. More generally, a proposed $\mu_0$ belongs to the unrounded 95% <confidence interval> precisely when its <two-sided test> does not reject under this strict cutoff. The equality boundary has <probability> zero in this continuous <statistical model>.
\Image[/hypothesis-test-normal-tails.png]
{title=The chosen 5% rejection tails and the observed two-sided p-value}
The <significance level> and <statistical power> describe different repeated-experiment <probabilities>. At a specified alternative $\mu$, put $\delta=(\mu-\mu_0)/(\sigma/\sqrt n)$ and $q=z_{1-\alpha/2}$. Then
$$
\begin{aligned}
\mathbb P_{\mu_0}(\text{reject }H_0)&=\alpha,\\
\beta(\mu)=\mathbb P_\mu(\text{do not reject }H_0)&=\Phi(q-\delta)-\Phi(-q-\delta),\\
\mathbb P_\mu(\text{reject }H_0)&=1-\beta(\mu).
\end{aligned}
$$
The first is the <Type I error> <probability>; the second is the <Type II error> <probability> at that alternative. For a planning alternative $\mu=1$, keeping $\mu_0=0$, $\sigma=2$ and $n=16$ gives $\delta=2$, $\beta(1)\approx0.483995$ and <statistical power> $0.516005$. At $n=64$, $\delta=4$ and <statistical power> rises to $0.979327$. These are model calculations for specified alternatives, not <posterior probabilities> inferred from a nonsignificant result. They explain why failure to reject cannot establish equality and why choosing <sample size> before an experiment is useful.
Historically, Ronald Fisher's 1925 book discusses the normal cutoff $1.96$ and tail <probability> 0.05 as a convenient convention on page 47, rather than a mathematically compulsory threshold: https://books.google.com/books/about/Statistical_Methods_for_Research_Workers.html?id=I0NBAAAAIAAJ[Fisher, Statistical Methods for Research Workers, first edition]{ref}.
Back to article page
