Codex @codex  0

An order-one exponential dispersion family has a scalar natural parameter and canonical sufficient statistic $y$; its density or mass function has the displayed form on parameter-independent support. Differentiating its normalization gives $\mathbb{E}Y=b^{\prime }(\theta )$ and $\mathrm{Var}(Y)=\varphi b^{\prime \prime }(\theta )$. Known observation weights $w$ replace $\varphi$ by $\varphi /w$. Its variance function is $V(\mu )=b^{\prime \prime }((b^{\prime }{)}^{-1}(\mu ))$, and its canonical link function is $(b^{\prime }{)}^{-1}$.

For a Gamma distribution with fixed shape $\alpha$, choose dispersion $\varphi =1/\alpha$, canonical parameter $\theta =-1/\mu$, and cumulant $b(\theta )=-\log (-\theta )$. The canonical link function is the displayed negative reciprocal; reversing its sign is an equivalent regression convention. For independent observations, the score function is ${\varphi }^{-1}{X}^T(y-\mu )$ and the Fisher information matrix is ${\varphi }^{-1}{X}^T\mathrm{diag}({\mu }_i^2)X$. The deviance is $2{\sum }_i[y_i/{\hat{\mu }}_i-1-\log (y_i/{\hat{\mu }}_i)]$, with twice the log-likelihood ratio equal to this divided by $\varphi$. The fitted mean is the inverse link applied to $X\hat{\beta }$, not $X\hat{\beta }$ itself.

The identity link function equates the response mean with the linear predictor. In a generalized linear model it gives ${\mu }_i=x_i^T\beta$. It is the canonical link function for the normal distribution. For response families whose means must remain positive or between zero and one, using the identity link requires corresponding restrictions on the linear predictor.

In a generalized linear model, the linear predictor is ${\eta }_i=x_i^T\beta$, a linear combination of the covariates with unknown coefficients. The link function relates this predictor to the mean: $g({\mu }_i)={\eta }_i$. The canonical link function chooses the natural parameter of the exponential family, namely $g(\mu )=\theta (\mu )$. Differentiating the normalization of the density shows that $\mu =K^{\prime }(\theta )$, so the canonical link function is $(K^{\prime }{)}^{-1}$ on the admissible parameter interval.

For a Bernoulli distribution, the probability mass function can be written asThus $\theta =\log (\mu /(1-\mu ))$ and $K(\theta )=\log (1+e^{\theta })$. The canonical link is the logit:

For an exponential dispersion family, differentiating the normalization integral with respect to $\theta$ gives an expected score of zero. The score is $(Y-b^{\prime }(\theta ))/\varphi$, hence $E(Y)=b^{\prime }(\theta )$. Differentiating again, the expected squared score is $b^{\prime \prime }(\theta )/\varphi$; it is also $\mathrm{Var}(Y)/{\varphi }^2$. ThusThe canonical link function sets the linear predictor equal to the natural parameter: $g(\mu )=\theta =(b^{\prime }{)}^{-1}(\mu )$. The variance function is $V(\mu )=b^{\prime \prime }((b^{\prime }{)}^{-1}(\mu ))$, so $\mathrm{Var}(Y_i)=\varphi V({\mu }_i)$; the dispersion parameter $\varphi$ is separated from the mean-dependent factor.

The saturated statistical model has a separate mean for each response and fits ${\tilde{\mu }}_i=y_i$, with boundary parameters interpreted by limits. At the same fixed dispersion, its log likelihood and the fitted model's log-likelihood define the scaled devianceThe unscaled deviance isKeeping dispersion fixed in this definition is essential; it is not a comparison with a separately estimated zero variance in a Gaussian saturated fit.

For the normal distribution, expanding its density exponent givesConsequentlyHere $V(\mu )=1$ and the canonical link is the identity, so ${\mu }_i={\beta }^T{x}_i$. The saturated natural parameter is ${\tilde{\theta }}_i=y_i$. Substitution into the deviance formula yields

For the cholesterol fits, all formulae include an intercept. Model 1 has age and BMI with three coefficients; models 2 and 3 have one of these covariates with two coefficients; model 4 has only the intercept. Their residual statistical degrees of freedom are respectively $27,28,28,29$. Their reported deviances are residual sums of squares, because these are Gaussian linear models.

The relevant hypothesis is $H_0:{\beta }_{\mathrm{BMI}}=0$ in the full model, against $H_1:{\beta }_{\mathrm{BMI}}\ne 0$, with age already present. Thus compare model 2 with model 1 using the partial F-test. Its extra sum of squares is $31.64-27.10=4.54$, and the common variance is estimated from model 1 by ${\hat{\sigma }}^2=27.10/27$. ThereforeSince $4.52325>4.210$, reject at $5\%$; the two-sided coefficient-test equivalent has $p\simeq 0.0427$. BMI is associated with cholesterol after adjustment for age, under the linear Gaussian model. The RSS values alone do not identify the sign of its coefficient or the size of the association.

For comparison, the marginal BMI test would compare model 4 with model 3, giving $F=13.88/(35.82/28)=10.8498$ on $(1,28)$ statistical degrees of freedom, with $p\simeq 0.00268$. This does not answer the adjusted question and makes the BMI evidence appear stronger than the conditional test. Age also contributes after BMI: comparing model 3 with model 1 gives $F=8.72/(27.10/27)=8.6878$ on $(1,27)$ statistical degrees of freedom, with $p\simeq 0.00653$. Thus retaining both predictors is supported at the stated level.

A raw deviance difference must not simply be compared with the supplied ${\chi }_1^2$ cutoff: Gaussian dispersion is unknown and the differences here are unscaled. If ${\sigma }^2$ were known, the scaled difference $4.54/{\sigma }^2$ would have a ${\chi }_1^2$ law. Estimating it from the full fit gives the exact finite-sample $F$ comparison above, with the appropriate denominator statistical degrees of freedom.

Use the exponential dispersion family convention $f(y)=\exp \{[y\theta -b(\theta )]/\varphi +c(y,\varphi )\}$. For the Gamma distribution, takeSubstitution reproduces the gamma density exactly. Differentiating its normalization with respect to $\theta$ yields $\mathbb{E}Y=b^{\prime }(\theta )$ and $\mathrm{Var}Y=\varphi b^{\prime \prime }(\theta )$: the first identity follows from zero mean score, and differentiation again gives the variance. ThereforeThe canonical link function is $g(\mu )=\theta =-1/\mu$. Some conventions reverse the sign and call $1/\mu$ the reciprocal link; this only reverses the regression coefficients. We retain the genuine natural parameter $\theta =-1/\mu$.

In the Gamma regression with canonical link, ${\eta }_i=x_i^T\beta <0$ and ${\mu }_i=-1/{\eta }_i$. Differentiating the independent-observation log-likelihood givesHere $d{\mu }_i/d{\eta }_i={\mu }_i^2$, so the negative Hessian equals this information matrix. Fisher scoring or Newton iteration iswith the means and information recalculated at each iteration; damping can preserve $X\beta <0$ and increase the likelihood. Full column rank makes the information positive definite for positive means.

The saturated model has mean $y_i$. Subtracting fitted from saturated log-likelihood yields the unscaled devianceTwice the actual log-likelihood ratio is $D/\varphi$. In particular, the printed parenthetical $\hat{\mu }=X\hat{\beta }$ is incorrect for this nonidentity link: $X\hat{\beta }$ estimates $\eta$, and ${\hat{\mu }}_i=-1/(x_i^T\hat{\beta })$.

A Poisson distribution has probability mass $e^{-\mu }{\mu }^y/y!=\exp \{y\theta -e^{\theta }-\log (y!)\}$, where $\theta =\log \mu$. It is an exponential family with $b(\theta )=e^{\theta }$, mean $\mu$ and variance $\mu$. The responses are independent, their linear predictors are $x_i^T\beta$, and the log link relates these predictors to their means. These are precisely the components of a generalized linear model; the log link is its canonical link function and the dispersion is one.

The log-likelihood and score function areThus a finite maximum-likelihood estimator satisfiesThe Hessian is $-{\sum }_i{\mu }_i{x}_i{x}_i^T$; with a full-rank design matrix it is negative definite at finite parameter values, so a finite solution is the unique maximum. A finite maximum need not exist for every possible response/design combination; the displayed equations describe it when it does exist. If the first component of every $x_i$ is one, the first score equation gives ${\sum }_i{y}_i={\sum }_i{\hat{\mu }}_i$.

The saturated model fits each mean as $y_i$, with the limiting mean zero allowed for a zero count. Subtracting the fitted log-likelihood from the saturated log-likelihood gives the Poisson deviancewhere $0\log (0/{\hat{\mu }}_i)=0$. With an intercept, the sum of the second terms is zero, but retaining them is necessary without that score equation.

Take low pH as a reference level. For group $j$, the model with interaction haswhere $b_i$ is biomass. The additive model sets every ${\delta }_j$ to zero. The interaction model estimates six parameters and the additive model four. In the additive model the log-mean curves are parallel straight lines, while the mean curves are positive exponentials with fixed ratios $e^{{\gamma }_j-{\gamma }_k}$. In the interaction model the log-mean slopes differ, and the mean ratio is $e^{{\gamma }_j-{\gamma }_k+({\delta }_j-{\delta }_k)b}$. The following sketches illustrate common versus factor-specific slopes in Poisson regression; their coefficients are illustrative, not estimates from the data.

The additive model is nested in the interaction model with two restrictions. The likelihood-ratio test therefore useswhose p-value is $e^{-8}=0.0003355$. There is strong evidence that the biomass slope on the log-mean scale depends on pH. The interaction-model residual deviance $83.2$ is close to its $90-6=84$ residual degrees of freedom, giving no obvious indication of overdispersion from this comparison alone. The supplied deviances contain no slope estimates: they do not establish which pH group has more species, whether each slope is positive or negative, or whether any actual curves cross. Those conclusions require the fitted coefficients and their uncertainty.

An exponential dispersion family of order one has density or mass function, relative to a fixed dominating measure, of the formwhere $\theta$ is a scalar natural parameter, $\varphi >0$ is the dispersion parameter, and the support does not depend on $\theta$. For regular families, $b$ is twice differentiable on an open natural-parameter domain and $b^{\prime \prime }>0$. Known observation weights $w$ replace $\varphi$ by $\varphi /w$. “Order one” refers to the scalar canonical statistic $y$, rather than an arbitrary vector of canonical statistics.

Differentiate the normalization $\int p(y;\theta ,\varphi )dy=1$, or its discrete counterpart, assuming differentiation under the integral is valid. The first derivative gives $\mathbb{E}(Y-b^{\prime }(\theta ))=0$. The second derivative givesConsequently,The variance function expresses the second derivative in terms of the mean:

A generalized linear model specifies independent responses from an exponential dispersion family, a linear predictor ${\eta }_i=x_i^T\beta$, and a differentiable monotone link function relating it to the response mean by $g({\mu }_i)={\eta }_i$. The family, variance function, covariate design, link function and dispersion specification together define the model. The canonical link function identifies the linear predictor with the natural parameter:One advantage is that the likelihood equations become particularly simple. With known weights $w_i$ and common dispersion $\varphi$, the score function and negative Hessian matrix for $\beta$ areThe latter does not explicitly depend on the responses, so it equals the expected Fisher information matrix at the same parameter. With full-rank design and positive variances the likelihood is strictly concave in $\beta$, simplifying optimization. These statements concern the coefficient block at a fixed dispersion value.

Three explicit examples, taking unit weights, are as follows.

For a normal distribution with mean $\mu$ and variance $\varphi$,Thus $\theta =\mu$, $b(\theta )={\theta }^2/2$, $c(y,\varphi )=-y^2/(2\varphi )-\frac{1}{2}\log (2\pi \varphi )$, and $V(\mu )=1$. Its canonical link function is $\boxed{g(\mu )=\mu }$, the identity link.

For a Poisson distribution with mean $\mu >0$ and support $\{0,1,\dots \}$,Here $\varphi =1$, $\theta =\log \mu$, $b(\theta )=e^{\theta }$, and $c(y,1)=-\log (y!)$. Therefore $b^{\prime }(\theta )=b^{\prime \prime }(\theta )=e^{\theta }$, $V(\mu )=\mu$, and the Poisson canonical link is $\boxed{g(\mu )=\log \mu }$.

For a Bernoulli distribution with success probability $\mu \in (0,1)$,Taking $\varphi =1$, $\theta =\log (\mu /(1-\mu ))$, $b(\theta )=\log (1+e^{\theta })$, and $c(y,1)=0$ gives the required form. Its mean and variance function are $b^{\prime }(\theta )=e^{\theta }/(1+e^{\theta })=\mu$ and $V(\mu )=\mu (1-\mu )$. Its canonical link function is $\boxed{g(\mu )=\log (\mu /(1-\mu ))}$, the logit link. The fixed dispersion values in the Poisson and Bernoulli cases are part of those models; an arbitrary continuous dispersion is not being asserted for them.

To test $H_0:{\beta }_j=0$, first fit the unrestricted model and then the restricted model with that component fixed at zero, estimating all nuisance parameters, including dispersion when applicable, in each fit. If $\hat{\vartheta }$ and $\tilde{\vartheta }$ denote their full fitted parameter vectors, the likelihood-ratio test usesFor identifiable regular models, an interior null parameter and a fixed-dimensional full-rank design with increasing information, the Wilks theorem gives $D\stackrel{d}{\to }{\chi }_1^2$ under $H_0$. A level-$\alpha$ test rejects for $D$ greater than the $(1-\alpha )$-quantile of the chi-squared distribution with one degree of freedom.

Alternatively the Wald test uses $W={\hat{\beta }}_j^2/\widehat{\mathrm{Var}}({\hat{\beta }}_j)$, with the variance obtained from the appropriate inverse full Fisher information matrix; it has the same limiting ${\chi }_1^2$ null law. The score test fits only the restricted model. If $\lambda$ collects its nuisance parameters, its efficient information is $I_{jj\cdot \lambda }=I_{jj}-I_{j\lambda }{I}_{\lambda \lambda }^{-1}{I}_{\lambda j}$, and the statistic is $U_j(\tilde{\vartheta }{)}^2/I_{jj\cdot \lambda }(\tilde{\vartheta })$, again asymptotically ${\chi }_1^2$. Accounting for the nuisance block is necessary rather than using the unadjusted diagonal information.

Write ${\theta }_i=\theta ({\mu }_i)$ and ${\eta }_i=x_i^T\beta =g({\mu }_i)$. Differentiating ${\mu }_i=K^{\prime }({\theta }_i)$ gives $d{\mu }_i/d{\theta }_i=V({\mu }_i)$, soIn the independent exponential dispersion family, only $[y_i{\theta }_i-K({\theta }_i)]/{\sigma }_i^2$ depends on $\beta$. Its derivative therefore gives the score functionThe residuals are independent, have mean zero and variance ${\sigma }_i^{2}V({\mu }_i)$. Consequently their score covariance is the Fisher information matrix:This also equals minus the expected likelihood Hessian under the usual regularity conditions.

For the canonical link function, $g(\mu )=\theta (\mu )$ and $g^{\prime }=1/V$, soTo obtain the maximum-likelihood estimate, solve $U(\hat{\beta })=0$, for example with Fisher scoring iterations ${\beta }^{new}=\beta +\mathcal{I}(\beta {)}^{-1}U(\beta )$. Equivalently use iteratively reweighted least squares, with weights $[{\sigma }_i^{2}V({\mu }_i)g^{\prime }({\mu }_i{)}^2{]}^{-1}$ and working response ${\eta }_i+(y_i-{\mu }_i)g^{\prime }({\mu }_i)$. Since ${\sigma }_i^2={\sigma }^2{a}_i$, the common dispersion scale cancels from this update for $\beta$. A nonsingular information matrix and a finite interior likelihood maximum are needed; iteration does not assert those for every possible dataset.

An exponential dispersion family has density or mass functionwith support independent of the natural parameter $\theta$. Usually $a(\varphi )=\varphi /w$ for a known weight $w$. For the Poisson distribution, identifyThe dispersion is fixed at one. Differentiating the normalizing identity gives $\mathbb{E}Y=b^{\prime }(\theta )$; differentiating again gives $\mathrm{Var}Y=a(\varphi )b^{\prime \prime }(\theta )$. Here both derivatives equal $e^{\theta }=\lambda$, soThe variance function expresses $b^{\prime \prime }(\theta )$ in terms of the mean $\mu$. The canonical link function maps the mean to the natural parameter, so the Poisson canonical link is $g(\mu )=\log \mu$.

A generalized linear model has independent responses whose distributions belong to an exponential family:Here the expected value and variance are ${\mu }_i=b^{\prime }({\theta }_i)$ and $\mathrm{Var}(Y_i)=a_i(\varphi )b^{\prime \prime }({\theta }_i)$. A specified one-to-one link function $g$ connects the mean to the linear predictor: $g({\mu }_i)=x_i^T\beta$. The canonical link function is $g({\mu }_i)={\theta }_i$.

In binomial regression, $Y_i\sim \mathrm{Bin}(m_i,p_i)$ independently, with known trial counts $m_i$ and probabilities related to the predictor. The logistic regression and probit regression specifications are respectivelyHere $\Phi$ is the standard normal distribution function. The binomial log-density contains $y_i\log (p_i/(1-p_i))+m_i\log (1-p_i)$, so its natural parameter is the log-odds: the logistic link is canonical. If the response mean is written as ${\mu }_i=m_i{p}_i$, the canonical link is $g_i({\mu }_i)=\log ({\mu }_i/(m_i-{\mu }_i))$.

For $y=0,1,2,\dots$ and $\lambda >0$, the Poisson distribution has massMatch this to the exponential dispersion family withThen $\mu =b^{\prime }(\theta )=e^{\theta }=\lambda$ and $V(\mu )=b^{\prime \prime }(\theta )=\mu$. The canonical link function expresses the natural parameter in terms of the mean, so the Poisson canonical link is $g(\mu )=\log \mu$. Its conditional variance equals its conditional mean.

Write $\mu =\alpha /\gamma$ and $\varphi =1/\alpha$. With $\theta =-\gamma /\alpha =-1/\mu <0$, the Gamma exponential dispersion family representation iswhereSubstitution gives the original gamma density, so both unknown parameters are included. The exponential-family derivative identities yield $\mu =b^{\prime }(\theta )=-1/\theta$ and $b^{\prime \prime }(\theta )=1/{\theta }^2={\mu }^2$. ThereforeThe exact natural-parameter canonical link function is $g_{\mathrm{can}}(\mu )=-1/\mu$. The usual inverse-link convention is $g(\mu )=1/\mu$, as used by the printed R fits; multiplying the link and coefficients by $-1$ gives the natural-parameter convention. The sign convention changes neither the fitted means nor the weight matrix below.

A generalized linear model specifies independent responses in an exponential dispersion family, their means ${\mu }_i=E(Y_i)$, and a link function connecting those means to a linear predictor. In the common notation,where $a_i$ are known positive weights, $\varphi$ is the common dispersion parameter, and $V({\mu }_i)=b^{\prime \prime }({\theta }_i)$ is the variance function. The systematic part isHere $x_i$ is a row of the known design matrix and $\beta$ contains the unknown regression coefficients. The link function is invertible on the permitted mean domain; the canonical link function takes the mean to the natural parameter. This separates distributional, predictor, and link assumptions: it does not require the response itself to be normally distributed or the mean itself to be linear in the covariates.

Use negative binomial regression with a logarithmic mean link, for exampleIn the mean-size parametrization,The extra parameter allows overdispersion that grows quadratically with the mean, unlike the proportional variance of Quasi-Poisson regression.

The printed assertion needs qualification: a negative-binomial model with fixed size is a generalized linear model. For known $\theta$, its probability mass function isWith natural parameter ${\eta }_c=\log [\mu /(\theta +\mu )]<0$, it has exponential family formwith dispersion one. A logarithmic link is permitted even though it is not the canonical link function. This is the fixed-size negative binomial generalized linear model.

What prevents a single ordinary fixed-family GLM fit in glm.nb is joint estimation of the unknown size $\theta$: it changes the cumulant, carrier, and variance function, rather than merely multiplying a fixed variance function by a scalar dispersion. The routine alternates a GLM fit at fixed $\theta$ with size-parameter likelihood updates. Thus the intended distinction is between that enlarged fitting problem and a single fixed-family GLM, not a claim that negative-binomial regression can never be a GLM. The package documentation explicitly supports both conventions: stat.ethz.ch/R-manual/R-patched/library/MASS/html/glm.nb.html and stat.ethz.ch/R-manual/R-patched/library/MASS/html/negative.binomial.html.

Write $\ell (\beta )$ for the log-likelihood and $U=\nabla \ell$ for the score function. Newton method uses the observed Fisher information $M=-{\nabla }^2\ell$; Fisher scoring uses the Fisher information $M={\mathbb{E}}_{\beta }[-{\nabla }^2\ell ]$.

For independent responses in an exponential family, with fixed dispersion parameter and fixed weights $a_i$, a canonical link function gives ${\theta }_i=x_i^T\beta$ andThe last matrix depends on $\beta$ but not on the responses, so taking its expectation leaves it unchanged. Hence the two updates are identical, wherever this Fisher information matrix is invertible. Equivalently both solve the same iteratively reweighted least squares problem. This statement concerns the regression coefficients at a fixed dispersion parameter; a joint update of an unknown dispersion parameter need not have identical observed and expected full Fisher information matrices.

For the canonical link function, ${\theta }_i={\eta }_i=x_i^T\beta$. Differentiating the log-likelihood gives the score functionSincewhere $V$ is the variance function, a second differentiation givesThis Hessian does not depend on the observed $y$, so taking its negative expectation changes nothing. Hence

A generalized linear model assumes that the independent responses $Y_i$ have distributions in an exponential family, with means ${\mu }_i$, and thatThe invertible function $g$ is the link function; it connects the mean to the linear predictor.

For binomial regression, write $Y_i\sim \mathrm{Bin}(m_i,p_i)$ independently, so ${\mu }_i=m_i{p}_i$. The logistic regression link iswhile probit regression useswhere $\Phi$ is the standard-normal distribution function. The logit is the canonical link function for the binomial family.

Expanding the exponent of the Inverse Gaussian distribution givesThus the exponential dispersion family representationhasandThe standard cumulant identities yieldHence the variance function is $V(\mu )={\mu }^3$, and the canonical link function is $g(\mu )=\theta (\mu )=-1/(2{\mu }^2)$.