Parameter estimation ($\hat{\theta }$)

Parameter estimation uses observations to infer the unknown parameters of a mathematical model. The result depends on both the observations and the model: one must specify which quantities are unknown, how observations relate to them, and how discrepancies are assessed. Least squares and maximum likelihood estimation provide two common criteria.^[ref]

For an explicit example, suppose $y_i=\theta +{\epsilon }_i$, where the same unknown level $\theta$ applies to all observations. MinimizinggivesSince $L^{\prime \prime }(\theta )=2n>0$, this is the unique minimum for $n>0$. Observations $2,4,6$ give the sample mean $\hat{\theta }=4$, with residuals $-2,0,2$. Independent normal distributions with a common variance also make this estimate the maximum likelihood estimation solution for the level.

An estimate is not the same as a known exact value. Under the independent equal-variance model, its standard error is estimated by $s/\sqrt{n}$, where $s^2$ is the sample variance. For the three illustrative observations, $s^2=4$ and the estimated standard error is $2/\sqrt{3}$. Correlated errors or a common measurement offset require different accounting; collecting more identically biased readings does not remove the offset.

Identifiability is a separate prerequisite. If a model depends only on $a+b$, data of that form can estimate their sum but cannot uniquely separate $a$ from $b$. Pairs $(1,3)$ and $(2,2)$ give the same sum. More repeated observations of that same relationship reduce uncertainty about the sum without identifying both parameters. State what the observations can determine before reporting numerical estimates.