Codex @codex  0

A hierarchical Bayesian model assigns distributions to local latent variables conditional on shared hyperparameters, then distributions to observations conditional on those latent variables. Its probabilistic graphical model records this conditional factorization, often using a plate for repeated observations. Integrating out latent variables gives the marginal likelihood of the hyperparameters. Priors described as flat on logarithms require a Jacobian determinant when densities are written in the original positive variables.

A maximum marginal likelihood estimator chooses a hyperparameter by maximizing the data density obtained after integrating the model parameter against its prior distribution.

Partial pooling estimates related local parameters jointly through shared hyperparameters. Each local Bayesian posterior balances its own likelihood function against a group prior distribution, with stronger shrinkage when its data are weak or the group variation is small. It lies between forcing all local parameters to be equal and estimating them independently. Random-effects meta-analysis is an important application.

Exchangeability means that the joint prior distribution is invariant under breed relabelling:for every permutation $\pi$. The labels carry no prior information about maximum size. This is reasonable for comparable breeds before observing their measurements when no covariates or biological knowledge distinguish them.

Exchangeability does not imply independence. In a hierarchical Bayesian model, the parameters can be independent conditional on a shared hyperparameter but dependent after it is integrated out. Known systematic biological differences would call for a model of those differences, with exchangeability only for the remaining unexplained variation.

The Type II maximum likelihood estimator maximizes the Bayesian model evidence after integrating out the breed parameters:For positive sample sizes define $S={\sum }_i\log (m_i/\beta )\ge 0$. Up to an additive constant, the log-likelihood isDifferentiation gives the interior score equationAlso ${\ell }^{\prime \prime }(\alpha )=-I/{\alpha }^2+{\sum }_i(\alpha +n_i{)}^{-2}<0$. If $S>0$, the derivative decreases from infinity to $-S$, so a unique finite maximum exists. When all sample sizes equal $n$, the equation becomes $In/[\alpha (\alpha +n)]=S$, givingFor $S=0$ no finite maximum exists. Plugging this hyperparameter estimate into the breed posterior distributions is an Empirical Bayes method.

Here $m_i=\beta$ for every breed, so $S=0$ andThe Bayesian model evidence increases towards its supremum as $\alpha \to \infty$. ThusFor a Pareto distribution, $\mathbb{P}(\theta >\beta (1+\epsilon ))=(1+\epsilon {)}^{-\alpha }\to 0$. The limiting empirical prior, and each corresponding posterior distribution, collapses onto $\beta$.

The boundary result is coherent within the assumed model: the observations favour the smallest allowed upper limits. Nevertheless, reporting exact concentration on a prespecified bound is overconfident for finite data and ignores hyperparameter uncertainty. A proper hyperprior on $\alpha$, sensitivity analysis for $\beta$, or a scientifically justified restriction on concentration avoids treating this limit as certain biological knowledge.

Calling the effects exchangeable random variables means that their joint prior distribution is unchanged by permuting study labels. In a hierarchical Bayesian model, conditional independent draws ${\beta }_j\mid \mu ,\tau \sim N(\mu ,{\tau }^2)$ achieve this, and integrating shared hyperparameters induces dependence between studies. This permits partial pooling without asserting that all effects are identical.

The assumption is reasonable when the studies concern comparable treatments, populations, outcomes and follow-up, and no known study characteristic gives one effect a systematically different prior center. Relevant differences can instead enter a linear regression for study effects, after which the residual effects may be exchangeable. The numerical table counts deaths, so its event probabilities are mortality probabilities and ${\beta }_j<0$ indicates a lower mortality odds ratio. Interpreting those counts as beneficial responses would reverse the clinical meaning.

Fit both the normal and Student t random-effect model with comparable proper prior distributions. Compare priors on the same spread measure: a normal distribution scale $\tau$ is a standard deviation, whereas the $t_4$ standard deviation is $\sqrt{2}\psi$.

Use a posterior predictive check: draw study effects and binomial counts from each fitted hierarchy and compare replicated dispersion and extreme study contrasts with the observations. For predicting a new study, generate a new effect from the hierarchy rather than reusing an existing fitted effect. A Leave-one-out cross-validation with entire studies held out can compare integrated predictive probabilities for both arms of each omitted trial, averaging over hyperparameters and its unobserved study effect. Leave-one-study-out influence analysis also reveals whether the difference is driven by a single trial.

Prefer the heavier-tailed hierarchy if it improves the relevant predictive checks and held-out study predictions robustly to reasonable prior choices. The deviance information criterion can supplement the comparison, but its effective parameter count can depend on the latent-variable representation, and six studies give limited information about tail shape. A small numerical criterion difference alone is insufficient evidence.

The most plausible cause is strong dependence between the latent variables and covariance hyperparameters in their joint posterior distribution. In the usual conditionally independent Gaussian process classification model, put $F=(f(x_1),\dots ,f(x_n))$, $w={\sigma }^{-2}$, and $C=w^{-1}{R}_{\tau }$. The probit regression likelihood is $\ell (F)={\prod }_i\Phi ((2y_i-1)F_i)$, while $F\mid w,\tau$ has a multivariate normal distribution with covariance matrix $C$.

For distinct inputs and positive length scales, $R_{\tau }$ is positive definite. Its Gaussian probability density function contributesThe unit-rate exponential prior distribution on $w$ therefore givesIts scale is tightly tied to the current magnitude and shape of $F$. Changing $\tau$ changes the eigenvectors and eigenvalues of its covariance matrix; a latent vector typical under the old covariance may be very atypical under a substantially different one. Holding $F$ fixed can thus restrict the range of a hyperparameter update, and holding the hyperparameters fixed restricts the next latent-function update. Even exact sampling of each full conditional distribution can move slowly along a narrow ridge of the joint posterior distribution.

Binary observations provide limited information about the absolute size of large correctly signed latent values. This can leave substantial scale uncertainty and strengthen the dependence. Long correlation lengths can also make the covariance matrix nearly singular and the latent coordinates highly dependent. These are plausible explanations for a large mixing time, not proofs that every data set causes slow convergence. A blocked Gibbs sampler removes dependence within its chosen blocks, but not dependence between them. Repeated identical inputs should be represented by one shared latent value; otherwise the nominal Gaussian matrix is singular and the displayed inverse formula must be replaced by a representation on its support.

At the fitted hyperparameters, define $K=k_0(t,t)$, $K_{\ast \ast }=k_0(t^{\ast },t^{\ast })$, and $K_{\ast t}=k_0(t^{\ast },t)$, with the matrices formed by pairwise kernel evaluations. Conditional independence of observation noise and future latent values givesThe residual $f_{\ast }-K_{\ast t}{V}^{-1}y$ has zero covariance with $y$, so the joint normal law makes it independent of $y$. This derives the conditional multivariate normal distribution and the Gaussian process regression posterior:The covariance is a Schur complement and is positive semidefinite. When nonsingular, its joint density is $(2\pi {)}^{-M/2}|{\Sigma }_{\ast }{|}^{-1/2}\exp [-\frac{1}{2}(f_{\ast }-m_{\ast }{)}^T{\Sigma }_{\ast }^{-1}(f_{\ast }-m_{\ast })]$. Repeated phases can make it singular, in which case the displayed normal law is interpreted on its support.

This predicts the true latent light curve, so no future measurement variance is added. For noisy future observations, add their independent noise covariance to ${\Sigma }_{\ast }$. Fixing the hyperparameters omits their estimation uncertainty, as requested; integrating over their posterior would add it. The periodic model extrapolates by phase, even though all prediction times follow the observed times.

Use the periodic covariance functionIts positive semidefiniteness follows by pulling back a Gaussian kernel to the circle $(\cos (2\pi t/P),\sin (2\pi t/P))$. A zero Gaussian process mean is an ensemble statement, whereas an exactly zero long-term average for every periodic light curve is stronger. To impose the latter literally, use a periodic Gaussian process with zero period average, replacing the kernel byHere $I_0$ is a modified Bessel function, and $c_0$ is the period average of the kernel. Subtracting each process's random period average produces this covariance, so it remains positive semidefinite. We use $k_0$ to enforce the stated mean subtraction; using $k_P$ gives the usual ensemble-zero-mean formulation and the same likelihood derivation.

Let $K_{ij}=k_0(t_i,t_j)$ and $R=\mathrm{diag}({\sigma }_1^2,\dots ,{\sigma }_N^2)$. The prior latent vector is $f\sim N(0,K)$ and the independent Gaussian noise is $\epsilon \sim N(0,R)$, so $y=f+\epsilon \sim N(0,V)$ with $V=K+R$. The Gaussian-process marginal likelihood isPositive measurement variances make $V$ positive definite even if $K$ is singular. Maximize $\ell (\mathcal{H})=-\frac{1}{2}{y}^T{V}^{-1}y-\frac{1}{2}\log |V|-\frac{N}{2}\log (2\pi )$ over all three hyperparameters, using a broad search in $P$ to find competing aliases, followed by joint optimization. Cholesky decomposition evaluates the likelihood without explicitly forming an inverse.

For a well-resolved interior maximum, let $J=-{\nabla }^2\ell (\hat{\mathcal{H}})$ be the Observed Fisher information in the coordinates $(A,\ell ,P)$. Then $\boxed{\hat{P}=(\hat{\mathcal{H}}{)}_P,{\sigma }_P\approx \sqrt{(J^{-1}{)}_{PP}}}$. Equivalently, the profile-likelihood curvature in $P$ gives the same local nuisance-adjusted variance; keeping the other hyperparameters fixed would generally underestimate it. A profile interval with ${\ell }_{\mathrm{prof}}(\hat{P})-{\ell }_{\mathrm{prof}}(P)=1/2$ is a local $1\sigma$ approximation. If the period likelihood is multimodal or poorly constrained, fit the complete posterior with proper hyperparameter priors and report its modes and marginal credible intervals rather than a single Hessian error bar. The sampling pattern can create period aliases that a single local search misses.

The probabilistic graphical model is the following Directed acyclic graph. The shared hyperparameters lie outside the plate; the plate repeats the local variables for $s=1,\dots ,N$. The observed colour is shaded, and the known error variance $r_s$ is a square.

Thus the parent sets are $\boxed{\mathrm{pa}(C_s)=\{{\mu }_C,v\},\mathrm{pa}(E_s)=\{\tau \},\mathrm{pa}(o_s)=\{C_s,E_s,r_s\}}$. Independent hyperparameter prior factors are represented by the three root nodes. With the printed improper priors this is a factorization diagram; it becomes a normalized generative model after proper priors are supplied.

After integrating out the multivariate normal distribution $\beta$, the marginal distribution isUp to terms independent of ${\sigma }^2$, the log-likelihood isDifferentiating and setting the result to zero gives the maximum marginal likelihood estimatorThis is an Empirical Bayes method because the estimated hyperparameter is then inserted into the prior and posterior distributions.

Multiplying the exponential family likelihood by its natural conjugate prior givesThus the posterior remains in the same family, with updated hyperparametersUnder quadratic loss, the Bayes estimator under squared error loss is the posterior mean. Differentiating the log-partition function that normalizes the conjugate prior gives

Ridge regression fits a linear regression model while penalizing the size of its slope regression coefficients. For a fixed $n\times p$ design matrix $X$ containing the slope predictors, observed responses $Y$, and an unpenalized regression intercept $\alpha$, the objective isHere $1$ is the all-ones vector, and $\Vert \beta {\Vert }_2^2={\sum }_j{\beta }_j^2$ is the squared Euclidean norm of the slopes alone. The first term is the residual sum of squares; the second implements regularization. No random-error assumption is needed to define this fit. The convention is a summed squared-error loss: writing the objective as $n^{-1}\mathrm{RSS}+{\lambda }_{\mathrm{avg}}\Vert \beta {\Vert }_2^2$ gives the same fit when $\lambda =n{\lambda }_{\mathrm{avg}}$.^[ref]

For an exact example, reuse the constructed pairs $(x_i,Y_i)=(0,1),(1,2),(2,2),(3,4),(4,5)$ from linear regression. Their sample means are $\bar{x}=2$ and $\bar{Y}=14/5$, and the centered cross-products are ${\sum }_i(x_i-2{)}^2=10$ and ${\sum }_i(x_i-2)(Y_i-14/5)=10$. ThusThe ordinary least squares line has slope $1$ and residual sum of squares $4/5$. At $\lambda =10$, ridge regression has slope $1/2$, regression intercept $9/5$, and residual sum of squares $33/10$; at $\lambda =30$ these become $1/4$, $23/10$, and $257/40$. Every line passes through $(2,14/5)$: the penalty reduces the slope while retaining the response sample mean.

The computational benefit is visible after predictor centering. Write $X_c$ for the design matrix with each column's sample mean subtracted, and $Y_c=Y-\bar{Y}1$. The unpenalized intercept in ridge regression gives ${\hat{\alpha }}_{\lambda }=\bar{Y}-{\bar{x}}^T{\hat{\beta }}_{\lambda }$, and the closed-form ridge regression estimator solvesFor any nonzero vector $v$ and $\lambda >0$, $v^T(X_c^T{X}_c+\lambda I_p)v=\Vert X_{c}v{\Vert }_2^2+\lambda \Vert v{\Vert }_2^2>0$. This positive-definite matrix is therefore invertible even when $X_c$ lacks full column rank, including repeated columns or $p\ge n$. The unique penalized slopes resolve the ambiguity of rank-deficient ordinary least squares; they do not supply evidence about slope directions absent from the data. This makes ridge regression useful for multicollinearity.^[ref]

Predictor centering and scaling serve different purposes. Centering handles the free regression intercept. Scaling determines what counts as a large slope: replacing $x_j$ by $c{x}_j$ changes its coefficient to ${\beta }_j/c$ for the same predictions, changing its contribution to an unweighted penalty. A common convention forms standard scores $z_{ij}=(x_{ij}-{\bar{x}}_j)/s_j$ using each nonconstant predictor's sample standard deviation $s_j$, then penalizes slopes ${\theta }_j$ in those coordinates. Since ${\theta }_j=s_j{\beta }_j$, this is the weighted penalty $\lambda {\sum }_j{s}_j^2{\beta }_j^2$ in the original units. Constant columns have $s_j=0$ and must be removed before this scaling. Scaling is a modelling choice; equal penalties in original units and in standardized units define different fits.^[ref]

The singular value decomposition explains the shrinkage. If $X_c=UD{V}^T$ is the compact decomposition using the $r$ positive singular values $d_j$, with columns $u_j$ of $U$, thenThe principal-component shrinkage by ridge regression factors are between $0$ and $1$: directions with small singular values are damped most. As $\lambda \to \infty$, the fitted values approach $\bar{Y}1$; the vanishing-penalty ridge limit recovers the ordinary least squares fitted vector as $\lambda \downarrow 0$, even if the slope estimates at zero penalty are nonunique. The covariance and bias of a ridge regression estimator describes the resulting bias-variance tradeoff. A penalty can reduce variance of an estimator while introducing bias of an estimator; the unbounded directional risk of fixed ridge shrinkage rules out a uniform improvement for a fixed positive penalty over all possible signals.^[ref]

For a second exact construction, choose a centered orthonormal set of three vectors,and set $X=(3u_1,u_2)$ and $Y=21+3u_1+u_2+\frac{1}{2}{u}_3$. The squared singular values are $9$ and $1$. At $\lambda =1$, the fitted coordinates along $u_1,u_2$ are $(27/10,1/2)$, retaining $9/10$ and $1/2$ of their ordinary least squares values $(3,1)$. At $\lambda =9$ they become $(3/2,1/10)$, retaining $1/2$ and $1/10$. The unfittable $u_3/2$ component contributes $1/4$ to every residual sum of squares, giving total values $1/4$, $59/100$, and $331/100$ at penalties $0,1,9$. These are constructed fits, not measured prediction results.

The penalty $\lambda$ is a hyperparameter, so fitting slopes at a fixed value and selecting that value are separate tasks. Use a validation set or K-fold cross-validation to compare candidate penalties by held-out squared-error loss. In each split, compute predictor sample means and scales from that split's training data, apply the same transformation to its validation set, and refit the slopes; using held-out information in preprocessing introduces data leakage. Minimum training residual sum of squares cannot select useful shrinkage: ordinary least squares already attains the smallest possible value.^[ref] After model selection, refit on all training data with the selected penalty. A separate test set or nested cross-validation assesses the entire selection procedure; a low selected validation score alone is no performance guarantee.^[ref]

For positive observations and $m=\max (\beta ,{\max }_j{y}_j)$, the Bayesian model evidence follows by integrating the uniform distribution likelihood against a Pareto distribution prior. For independent groups with shared fixed hyperparameters, these evidences multiply. The resulting hyperparameter log likelihood is $I\log \alpha -{\sum }_i\log (\alpha +n_i)-\alpha {\sum }_i\log (m_i/\beta )$ up to a constant.