Generalized linear model

A generalized linear model takes independent responses from an exponential dispersion family and relates their means to linear predictors by $g({\mu }_i)=x_i^T\beta$. The response distribution, systematic component, link, dispersion, and known observation weights together specify the model.

For a suitably regular generalized linear model, a residual deviance $D$ can be compared approximately with a chi-squared distribution on $n-p$ degrees of freedom to assess fit relative to the saturated model. Sparse or very small expected cell counts can invalidate this reference distribution. A large p-value means no detected lack of fit, not proof of correctness; dependence, wrong mean structure or overdispersion still require diagnostics.

A deviance residual is the signed square root of an observation's contribution $d_i$ to fitted-model deviance. For a Poisson regression, $d_i=2[y_i\log (y_i/{\hat{\mu }}_i)-(y_i-{\hat{\mu }}_i)]$, with $0\log 0=0$. Its squared sum is the Poisson deviance. Leverage and estimated dispersion can further standardize these residuals; a normal quantile-quantile plot is a diagnostic approximation, not a requirement that count errors have a normal distribution.

A generalized additive model has linear predictor $g({\mu }_i)=\alpha +{\sum }_j{s}_j(x_{ij})$. Centering the smooth functions identifies the intercept. The response family and link function remain part of the model; an unspecified software family must not be inferred from the response name.

The backfitting algorithm estimates each additive mean function by smoothing its partial residual, obtained after subtracting the current fits of all other terms, and cycling through terms until convergence. Centering each smooth separates its constant part from the intercept. For fixed quadratic smoothness penalties it is block coordinate minimization of a penalized least-squares objective; identifiability and positive definiteness of the constrained problem ensure a unique converged fit. Non-Gaussian generalized additive models can use weighted backfitting inside iteratively reweighted least squares.

A generalized additive mixed model supplements smooth population effects with latent random effects. Given these effects, responses follow a specified response family; integrating them out induces dependence between related observations.

Quasi-likelihood specifies a mean and a variance function without necessarily specifying a full response distribution. For independent responses, the estimating equation is ${\sum }_i(\partial {\mu }_i/\partial \beta )(y_i-{\mu }_i)/(\varphi V({\mu }_i))=0$. A common scalar dispersion parameter rescales coefficient covariance while leaving the roots of the quasi-score equation unchanged.

The logit mean model $p_i=(1+e^{-x_i^T\beta }{)}^{-1}$ with working variance function $\varphi p_i(1-p_i)$ has the same coefficient estimates as ordinary binomial logistic regression, but estimates dispersion from residuals. For an actual individual binary random variable, $Y^2=Y$ forces variance $p(1-p)$, so a nonunit dispersion is a working specification rather than a new independent binary distribution.

The linear predictor of a generalized linear model is the linear combination ${\eta }_i=x_i^T\beta$ of the explanatory variables.

A link function connects the response mean to the linear predictor through $g({\mu }_i)={\eta }_i$.

The logit link maps a mean probability to its log odds: $g(p)=\log [p/(1-p)]$, $0<p<1$. Its inverse is $p=(1+e^{-\eta }{)}^{-1}$, used in logistic regression.

The logarithmic link maps a positive mean to $\log \mu$. It is the Poisson canonical link.

Binomial regression models independent $Y_i\sim \mathrm{Bin}(m_i,p_i)$ and relates $p_i$ to a linear predictor. Logistic and probit regression use the logit and inverse-normal links respectively.

For independent binomial counts $Y_i\sim \mathrm{Bin}(m_i,p_i)$ with observed proportions $y_i=Y_i/m_i$ and fitted probabilities ${\hat{p}}_i$, the deviance from the saturated model iswith the usual convention $0\log 0=0$.

For a generalized linear model with variance function $V$, the Pearson estimator is

The Pearson residual is $r_i^P=(Y_i-{\hat{\mu }}_i)/\sqrt{\hat{\varphi }V({\hat{\mu }}_i)}$.

The Pearson chi-squared statistic for a generalized linear model isDividing it by the residual degrees of freedom estimates the dispersion parameter.

The canonical link identifies the linear predictor with the exponential family's natural parameter: $g(\mu )=\theta$, where $\mu =b^{\prime }(\theta )$.

For a Poisson mean $\mu$, the natural parameter is $\theta =\log \mu$, so the canonical link is the logarithmic link $g(\mu )=\log \mu$.

A Poisson regression models independent count responses byIts likelihood is ${\prod }_i{e}^{-{\mu }_i}{\mu }_i^{y_i}/y_i!$.

For three distinct times, the columns $1,t,t^2$ are linearly independent by the Vandermonde determinant. Thus a quadratic Poisson regression for the log mean can reproduce any three positive time-group means. A grouped goodness-of-fit test of that mean form has zero residual degrees of freedom. Replicated observations still allow checks of the Poisson distribution and overdispersion within each time group; saturation of the mean does not establish the response distribution.

A categorical covariate can be grouped in an interaction term while retaining its original main effect. This shares selected slopes but keeps separate intercepts. Nested Poisson deviance differences test slope pooling and intercept pooling separately.

In a Poisson regression with $\log \mu ={\beta }_0+{\beta }_{T}T+{\beta }_{B}b+{\beta }_{TB}Tb$, the treatment logarithm of the mean ratio at baseline $b$ is ${\beta }_T+b{\beta }_{TB}$. The main treatment coefficient tests the contrast at $b=0$. An overall no-treatment test requires both ${\beta }_T=0$ and ${\beta }_{TB}=0$, with their joint covariance matrix for a Wald test.

For independent Poisson observations with fitted means ${\hat{\mu }}_i$, the deviance from the saturated model iswith $0\log 0=0$.

A Gamma regression with logarithmic link models a positive response with $\mathrm{Var}(Y_i\mid X_i)=\varphi {\mu }_i^2$ and $\log {\mu }_i=x_i^T\beta$.

Quasi-Poisson regression uses the mean model $\log {\mu }_i=x_i^T\beta$ and variance relation $\mathrm{Var}(Y_i)=\varphi {\mu }_i$. It estimates $\beta$ through quasi-likelihood and scales its covariance by the estimated dispersion $\varphi$.

For mean ${\mu }_i(\beta )$ and variance proportional to $V({\mu }_i)$, the quasi-score is the sum of $(Y_i-{\mu }_i)V({\mu }_i{)}^{-1}\partial {\mu }_i/\partial \beta$. Setting it to zero estimates $\beta$ without requiring a full response likelihood.

Negative binomial regression models overdispersed counts with a negative binomial distribution, commonly using a logarithmic link for the mean.

For known size $\theta >0$, the negative binomial distribution is an exponential family with natural parameter $\eta =\log [\mu /(\theta +\mu )]<0$ and cumulant $b_{\theta }(\eta )=-\theta \log (1-e^{\eta })$. A generalized linear model can use a logarithmic link function for its mean, even though that link is not canonical. Jointly estimating the size changes the family and variance function, requiring additional updates beyond a single fixed-family GLM fit; this is what MASS::glm.nb does.

For fixed exposures $x_i\ge 0$, a Poisson exposure model takes independent observationsIt has mean and variance $\theta x_i$ and is therefore heteroscedastic when the exposures differ.

For $S_r={\sum }_i{x}_i^r$, unweighted least squares through the origin and maximum likelihood giveBoth estimators are unbiased.

The two variances areSince $S_2^2\le S_1{S}_3$ by Cauchy-Schwarz, maximum likelihood has no larger variance, with equality when all positive exposures are equal.

For testing $H_0:\theta =1$ against a larger alternative, approximate size-$0.05$ upper-tail tests reject whenor, respectively,

Logistic regression models the conditional log odds as an affine function of predictors.

A logistic regression can specify each binary response conditionally on recorded past outcomes and baseline covariates. A chain rule for probabilities gives the path likelihood as the product of the conditional Bernoulli distribution masses. This permits dependence within a person's history without treating their outcomes as unconditionally independent. The model must specify the initial history and which past features enter its conditional mean.

The accumulated number of previous successes may enter the conditional mean in a history-dependent logistic regression. Its coefficient exponentiates to the conditional success odds ratio per previous success. For a specified future path, update this cumulative predictor after each outcome before multiplying the conditional probabilities.

A previous outcome may affect a subsequent outcome after controlling both measured covariates and persistent unobserved heterogeneity. This is true state dependence. An observed lagged-outcome association alone does not establish it, because omitted individual propensities can create apparent persistence.

A regression for a categorical response with more than two possible outcomes, parametrizing log probability ratios relative to a baseline category by linear predictors. It preserves categories that a binary logistic regression would merge.

A logistic-normal count model takes $Y_i\mid {\theta }_i\sim \mathrm{Bin}(n_i,{\mathrm{logit}}^{-1}{\theta }_i)$ and ${\theta }_i=x_i^T\beta +\lambda Z_i+{\eta }_i$, where $Z$ is a stationary unit-variance autoregressive process of order one with coefficient $a$ and the independent ${\eta }_i$ have normal distribution with variance $v$. The marginal latent variance is ${\lambda }^2+v$, and its off-diagonal covariance is ${\lambda }^2{a}^{|i-j|}$. Random success probabilities produce overdispersion relative to the binomial distribution.

A logistic model maps a real-valued score $\eta$ to the probability $p=(1+e^{-\eta }{)}^{-1}$, equivalently $\log (p/(1-p))=\eta$. Logistic regression takes $\eta$ to be an affine function of the predictors.

For an event of probability $p$, the log odds are the natural logarithm of its odds:

Separation in a binary regression model occurs when an affine score assigns all observations from one class to one side of a threshold and all observations from the other class to the other side.

Complete separation means that every signed training margin is strictly positive for some affine score. In unpenalized logistic regression it causes coefficient norms to diverge and prevents a finite maximum-likelihood estimate.

For binary independent responses, logistic regression sets

At an interior maximum-likelihood fit of a logistic-regression model containing an intercept, the intercept score function isConsequently the sum of the fitted probabilities equals the number of observed successes exactly.

For a signed margin $u=yh(x)$, the logistic loss isIt is convex and has derivative ${\varphi }^{\prime }(u)=-e^{-u}/(1+e^{-u})$.

For a positive semidefinite trace ball ${\mathcal{S}}_s$, the quadratic-form hypothesis class isIts empirical logistic risk is convex as a function of $M$.

L1-penalized logistic regression minimizes empirical logistic loss plus $\lambda \Vert \beta {\Vert }_1$. Its subgradient optimality conditions set each logistic score coordinate equal to $-\lambda z_j$, where $z_j=\mathrm{sgn}({\beta }_j)$ off zero and $z_j\in [-1,1]$ at zero.

For independent grouped counts $Y_i\sim \mathrm{Bin}(n_i,p_i)$, grouped-binomial logistic regression specifiesFitting the proportions $Y_i/n_i$ with binomial weights $n_i$ gives the same likelihood.

With treatment coding, the intercept describes the reference factor level and each displayed factor coefficient is a contrast against that level. Changing the reference level reparametrizes the same fitted model: fitted probabilities and likelihood do not change, but coefficients representing different pairwise contrasts can have different standard errors and p-values.

A stochastic block model assigns each vertex of a graph to a class and, conditional on those classes, makes distinct edges independent random variables whose probabilities depend only on the classes of their endpoints.

For independent Bernoulli distribution upper-triangular entries of a symmetric zero-diagonal adjacency matrix of a graph, put $W=A-\mathbb{E}A$. For a unit vector $x$, $x^{\top }Wx=2{\sum }_{i<j}{x}_i{x}_j{W}_{ij}$ has sub-Gaussian variance proxy at most ${\sum }_{i<j}{x}_i^2{x}_j^2\le 1/2$, by the Hoeffding lemma. Hence $\mathbb{P}(|x^{\top }Wx|>u)\le 2e^{-u^2}$. The volumetric bound for Euclidean metric nets gives a $1/4$-metric net of the unit sphere with at most $9^n$ points. The quadratic form net bound gives $\Vert W{\Vert }_{\mathrm{op}}\le 2{\max }_x|x^{\top }Wx|$ on that net. The union bound gives $\mathbb{P}(\Vert W{\Vert }_{\mathrm{op}}>2\sqrt{n\log 9+\log 2+s})\le e^{-s}$; integrating this tail proves the displayed expectation bound.

A logistic stochastic block model parameterizes the edge probability between classes $k$ and $\ell$ as ${\mathrm{logit}}^{-1}({\beta }_{k\ell })$.

An additive class-effect logistic network model gives an edge joining classes $k$ and $\ell$ the log odds ${\beta }_k+{\beta }_{\ell }$. A further coefficient multiplying $1_{\{k=\ell \}}$ represents a common within-class log-odds effect.

For independent binary edges with log odds ${\beta }_{z_i}+{\beta }_{z_j}+{\beta }_0{1}_{\{z_i=z_j\}}$, the natural statistics are the sum of the vertex degrees in each class and the total number of within-class edges. They form a minimal sufficient statistic whenever the natural parameter space contains an open subset of ${\mathbb{R}}^{C+1}$.

Binary probit regression sets $p_i=\Phi ({\beta }_0+x_i^T\beta )$, where $\Phi$ is the standard normal distribution function. Its intercept score is a weighted sum of residuals, so an intercept does not generally force ${\sum }_i{\hat{p}}_i={\sum }_i{y}_i$.

For binary observations with success probability $\Phi (\beta x_i)$, introduce latent $Z_i\sim N(\beta x_i,1)$ and let the sign determine the observation. Given $\beta$ and the observed sign, each latent variable has a truncated normal distribution. With a standard normal prior, the full conditional distribution of $\beta$ is normal with precision $1+{\sum }_i{x}_i^2$ and mean ${\sum }_i{x}_i{Z}_i/(1+{\sum }_i{x}_i^2)$. Alternating these blocks is Gibbs sampling with the required marginal posterior.

Observing only whether a lognormally modelled output exceeds a known threshold produces a probit link. The latent location and slope are recovered by multiplying the probit coefficients by the known Gaussian scale and adding the log threshold to the intercept.

For probit regression with $s_i=2Y_i-1$ and prior $N(0,{\sigma }^{2}I)$, the posterior score control variate isFor $h(\beta )=\Phi (x_{\ast }^T\beta )$, its covariance with the score is $-x_{\ast }\mathbb{E}[\varphi (x_{\ast }^T\beta )]$. It is nonzero whenever $x_{\ast }\ne 0$, ensuring strict improvement by the optimal control variate.

IRLS fits a generalized linear model by repeatedly solving a weighted least-squares approximation to its score equations.

A generalized linear mixed model adds latent random effects to the linear predictor of a generalized linear model.

A Gaussian linear mixed model combines fixed effects, random effects with a normal distribution, and independent Gaussian noise. For $b\sim N(0,D)$ and $\epsilon \sim N(0,R)$ independently, its marginal distribution is $Y\sim N(X\beta ,V)$ with $V=ZD{Z}^T+R$. The random effects induce dependence among observations sharing their design columns. A random-intercept linear mixed model and a random-slope linear mixed model are special cases.

A subject-specific random intercept and random slope may have a joint bivariate normal distribution with arbitrary positive-definite covariance matrix $D$. With independent normal measurement errors of variance ${\sigma }^2$, the marginal covariance between a person's readings at $s,t$ is $D_{00}+(s+t)D_{01}+st{D}_{11}$, plus ${\sigma }^2$ on the diagonal. Different subjects remain independent. Allowing $D_{01}$ distinguishes this model from an independent random-intercept and random-slope model.

A variance component is a nonnegative parameter multiplying a specified covariance contribution. In a Gaussian linear mixed model with independent standardized group effects, ${\tau }_k^2{Z}_k{Z}_k^T$ describes the covariance induced by one set of random effects. Testing whether a component is zero is a variance-component likelihood-ratio test at a boundary; ordinary regular chi-squared likelihood-ratio calibration need not apply.

A method-of-moments variance component estimate equates residual mean squares in ANOVA to their model expectations and solves for the variance components. An unconstrained estimate can be negative even when the model component must be nonnegative; boundary treatment then requires a stated estimation convention. The method gives point estimates, not certainty about future experiments.

With known covariance parameters, best linear unbiased prediction minimizes prediction-error variance over linear predictors that are unbiased over both observation errors and random effects, for every fixed-effect value. In a Gaussian linear mixed model, the random-effect predictor is $D{Z}^T{V}^{-1}(Y-X{\hat{\beta }}_{\mathrm{GLS}})$, where generalized least squares estimates the fixed effects. Plugging in covariance estimates gives an empirical predictor; its uncertainty must also account for estimating those covariance parameters. It differs from a best linear unbiased estimator of an unknown fixed coefficient.

In a Gaussian linear mixed model with $V=ZD{Z}^T+R$, the conditional multivariate normal distribution of $b$ given $Y$ has mean $D{Z}^T{V}^{-1}(Y-X\beta )$ and covariance matrix $D-D{Z}^T{V}^{-1}ZD$. When nonsingular, its mean is also its mode. Estimated parameters yield empirical conditional modes and shrink group deviations toward zero. At a zero variance component the associated effect is degenerate at zero; the covariance formula still applies, whereas formulas involving $D^{-1}$ require limits.

For $\kappa >0$, this error correlation decreases exponentially with elapsed time and permits irregular observation times. It is the stationary correlation of an Ornstein-Uhlenbeck process. Within a Gaussian linear mixed model, apply it to the errors conditional on random effects; the marginal correlation also includes those effects. nlme::corCAR1 parametrizes the same correlation by $\rho =e^{-\kappa }\in (0,1)$ and permits separate grouped time series.

Independent Gaussian random intercepts $u_j$ and random slopes $v_j$ give marginal within-group covariance ${\tau }_0^2+{\tau }_1^2{t}_{ij}{t}_{kj}+{\sigma }^2{1}_{\{i=k\}}$. In lme4, separate terms (1 | group) and (0 + x | group) impose zero intercept-slope covariance; (1 + x | group) estimates it. This independence restriction depends on the predictor origin: replacing $t$ by $t-c$ transforms the intercept effect to $u_j+c{v}_j$, generally correlated with $v_j$.

A Poisson generalized linear mixed model gives a count response a conditional Poisson distribution whose logarithmic mean contains both fixed effects and latent random effects.

A random intercept gives every group a latent additive shift, inducing dependence among observations from the same group.

A random-intercept linear mixed model has the formwhere independent group effects $b_i$ induce positive covariance among observations from the same group.