Generalized linear model

A generalized linear model takes independent responses from an exponential dispersion family and relates their means to linear predictors by $g({\mu }_i)=x_i^T\beta$. The response distribution, systematic component, link, dispersion, and known observation weights together specify the model.

For a Gamma distribution with fixed shape $\alpha$, choose dispersion $\varphi =1/\alpha$, canonical parameter $\theta =-1/\mu$, and cumulant $b(\theta )=-\log (-\theta )$. The canonical link function is the displayed negative reciprocal; reversing its sign is an equivalent regression convention. For independent observations, the score function is ${\varphi }^{-1}{X}^T(y-\mu )$ and the Fisher information matrix is ${\varphi }^{-1}{X}^T\mathrm{diag}({\mu }_i^2)X$. The deviance is $2{\sum }_i[y_i/{\hat{\mu }}_i-1-\log (y_i/{\hat{\mu }}_i)]$, with twice the log-likelihood ratio equal to this divided by $\varphi$. The fitted mean is the inverse link applied to $X\hat{\beta }$, not $X\hat{\beta }$ itself.

An offset is a known term in a generalized linear model's linear predictor, with coefficient fixed to one. With a log link and exposure $n_i$, choosing $o_i=\log n_i$ gives a mean count $n_i\exp (x_i^T\beta )$. Its role differs from that of a fitted explanatory-variable coefficient.

For a suitably regular generalized linear model, a residual deviance $D$ can be compared approximately with a chi-squared distribution on $n-p$ degrees of freedom to assess fit relative to the saturated model. Sparse or very small expected cell counts can invalidate this reference distribution. A large p-value means no detected lack of fit, not proof of correctness; dependence, wrong mean structure or overdispersion still require diagnostics.

A logistic regression fitted to ungrouped Bernoulli observations has residual deviance $D=-2\ell (\hat{\beta })$. Comparing it automatically with ${\chi }_{n-p}^2$ is generally invalid: the saturated dimension grows with $n$, with one observation per fitted probability. If the true model has only an intercept and success probability $1/2$, then $D/n\to 2\log 2$, whereas ${\chi }_{n-1}^2/n\to 1$. Fixed-dimensional nested-model likelihood-ratio tests can still use Wilks theorem under regularity. For absolute fit, use meaningful grouping, residual diagnostics, or a fitted-model parametric bootstrap.

A deviance residual is the signed square root of an observation's contribution $d_i$ to fitted-model deviance. For a Poisson regression, $d_i=2[y_i\log (y_i/{\hat{\mu }}_i)-(y_i-{\hat{\mu }}_i)]$, with $0\log 0=0$. Its squared sum is the Poisson deviance. Leverage and estimated dispersion can further standardize these residuals; a normal quantile-quantile plot is a diagnostic approximation, not a requirement that count errors have a normal distribution.

A generalized additive model has linear predictor $g({\mu }_i)=\alpha +{\sum }_j{s}_j(x_{ij})$. Centering the smooth functions identifies the intercept. The response family and link function remain part of the model; an unspecified software family must not be inferred from the response name.

An additive regression model represents a conditional mean as a sum of univariate functions and an intercept. It permits nonlinear effects without estimating a general high-dimensional response surface. The backfitting algorithm smooths each component's partial residual. Centering separates constants from the intercept, while concurvity can still prevent component identification.

// Target: probability-and-statistics.bigb

In an additive regression model, each component can transfer a constant to the regression intercept, so impose empirical mean-zero constraints, or population mean-zero constraints when appropriate. Even after centering, nonzero component perturbations $h_j$ with ${\sum }_j{h}_j(x_{ij})=0$ at every observed point make the additive decomposition nonidentifiable. Extra smoothing penalties may resolve the fitted decomposition, but centering alone does not remove concurvity.

// Target: probability-and-statistics.bigb

Concurvity is the nonlinear analogue of multicollinearity: a function of one predictor is exactly or approximately expressible as a sum of functions of other predictors. Exact concurvity can make an additive decomposition nonunique; approximate concurvity makes components difficult to estimate stably. A unique fitted sum need not imply unique individual components.

// Target: probability-and-statistics.bigb

The backfitting algorithm estimates each additive mean function by smoothing its partial residual, obtained after subtracting the current fits of all other terms, and cycling through terms until convergence. Centering each smooth separates its constant part from the intercept. For fixed quadratic smoothness penalties it is block coordinate minimization of a penalized least-squares objective; identifiability and positive definiteness of the constrained problem ensure a unique converged fit. Non-Gaussian generalized additive models can use weighted backfitting inside iteratively reweighted least squares.

For linear smoothers, let $C=I-11^T/n$ and $T_j=C{S}_{j}C$. The centered fixed-point equations of the backfitting algorithm have diagonal identity blocks and off-diagonal blocks $T_j$. On the centered component space, a consistent system has a unique answer exactly when its homogeneous system has only the zero solution. For two components the reduced equation is $(I-T_1{T}_2)f_1=T_1(I-T_2)Cy$. A common centered vector reproduced by both smoothers yields opposite perturbations of the components without changing their sum.

// Target: probability-and-statistics.bigb

A generalized additive mixed model supplements smooth population effects with latent random effects. Given these effects, responses follow a specified response family; integrating them out induces dependence between related observations.

Quasi-likelihood specifies a mean and a variance function without necessarily specifying a full response distribution. For independent responses, the estimating equation is ${\sum }_i(\partial {\mu }_i/\partial \beta )(y_i-{\mu }_i)/(\varphi V({\mu }_i))=0$. A common scalar dispersion parameter rescales coefficient covariance while leaving the roots of the quasi-score equation unchanged.

The logit mean model $p_i=(1+e^{-x_i^T\beta }{)}^{-1}$ with working variance function $\varphi p_i(1-p_i)$ has the same coefficient estimates as ordinary binomial logistic regression, but estimates dispersion from residuals. For an actual individual binary random variable, $Y^2=Y$ forces variance $p(1-p)$, so a nonunit dispersion is a working specification rather than a new independent binary distribution.

The linear predictor of a generalized linear model is the linear combination ${\eta }_i=x_i^T\beta$ of the explanatory variables.

A link function connects the response mean to the linear predictor through $g({\mu }_i)={\eta }_i$.

The complementary log-log link maps $p\in (0,1)$ to $\ln [-\ln (1-p)]$, with inverse $p=1-\exp (-e^{\eta })$. For constant infection hazard $\lambda$ and exposure $e$, $p=1-e^{-\lambda e}$ implies $\mathrm{cloglog}(p)=\ln \lambda +\ln e$. Thus a binomial regression with a log-exposure offset estimates the log hazard as its intercept.

The identity link function equates the response mean with the linear predictor. In a generalized linear model it gives ${\mu }_i=x_i^T\beta$. It is the canonical link function for the normal distribution. For response families whose means must remain positive or between zero and one, using the identity link requires corresponding restrictions on the linear predictor.

The logit link maps a mean probability to its log odds: $g(p)=\log [p/(1-p)]$, $0<p<1$. Its inverse is $p=(1+e^{-\eta }{)}^{-1}$, used in logistic regression.

The logarithmic link maps a positive mean to $\log \mu$. It is the Poisson canonical link.

Binomial regression models independent $Y_i\sim \mathrm{Bin}(m_i,p_i)$ and relates $p_i$ to a linear predictor. Logistic and probit regression use the logit and inverse-normal links respectively.

For independent binomial counts $Y_i\sim \mathrm{Bin}(m_i,p_i)$ with observed proportions $y_i=Y_i/m_i$ and fitted probabilities ${\hat{p}}_i$, the deviance from the saturated model iswith the usual convention $0\log 0=0$.

For a generalized linear model with variance function $V$, the Pearson estimator is

The Pearson residual is $r_i^P=(Y_i-{\hat{\mu }}_i)/\sqrt{\hat{\varphi }V({\hat{\mu }}_i)}$.

The Pearson chi-squared statistic for a generalized linear model isDividing it by the residual degrees of freedom estimates the dispersion parameter.

The canonical link identifies the linear predictor with the exponential family's natural parameter: $g(\mu )=\theta$, where $\mu =b^{\prime }(\theta )$.

For a Poisson mean $\mu$, the natural parameter is $\theta =\log \mu$, so the canonical link is the logarithmic link $g(\mu )=\log \mu$.

A Poisson regression models independent count responses byIts likelihood is ${\prod }_i{e}^{-{\mu }_i}{\mu }_i^{y_i}/y_i!$.

For independent counts with means $p_{i1}{\lambda }_i$ and $p_{i2}{\lambda }_{i}r$, let $t_i=y_{i1}+y_{i2}$. Maximizing the Poisson regression likelihood in each baseline rate gives the displayed estimate. The remaining log-rate-ratio score isEquivalently, conditional on $t_i$, the second count has a binomial distribution with success probability $r{p}_{i2}/(p_{i1}+r{p}_{i2})$. Each positive-total term increases strictly with $r$; if both treatment totals are positive, the score has a unique positive root. This reduces a many-parameter fit to a scalar equation.

For a full-column-rank design $X$ and positive fitted means ${\mu }_i=\exp (x_i^T\beta )$, the log-likelihood has Hessian $-X^T\mathrm{diag}({\mu }_i)X$, which is negative definite. Thus any finite stationary point is the unique maximum-likelihood estimator. Full rank does not guarantee existence: with an intercept and all counts zero, the supremum is approached as the intercept tends to $-\infty$. If every observed count is positive, a full-rank design does give a finite maximum: each function $y_i{\eta }_i-e^{{\eta }_i}$ tends to $-\infty$ in either tail and is bounded above, and $\Vert X\beta \Vert \to \infty$ as $\Vert \beta \Vert \to \infty$, so the negative log-likelihood is coercive.

In Poisson regression with log means $a+\beta x_i$, the Fisher information matrix is $\left(\begin{array}{cc}{S}_0& S_1\\ S_1& S_2\end{array}\right)$, where $S_j={\sum }_i{x}_i^j{\mu }_i$. Inverting this matrix gives slope variance $S_0/(S_0{S}_2-S_1^2)$. Writing ${\bar{x}}_{\mu }=S_1/S_0$ reduces it to the displayed reciprocal weighted sum of squares. Thus a common intercept consumes information about the overall rate, while slope information comes from the spread of the covariates under the mean-count weights. Constant covariates make the two parameters unidentifiable.

For independent Poisson distributions with log means $Z\beta +o$, where $o$ is a known offset, differentiation of the log-likelihood gives the score function $Z^T(y-\mu )$. An interior maximum-likelihood estimator therefore satisfies the displayed equations. An intercept and categorical indicator columns make fitted count totals equal observed count totals in each represented factor level. These equations match counts, rather than unweighted averages of rates when exposures differ.

// Target: probability-and-statistics.bigb

For independent aggregate Poisson distributions with means $n\lambda$ and $n\lambda e^{\beta }$, conditioning on the total eliminates $\lambda$. Independent Poisson conditioning gives the displayed binomial distribution. Its conditional likelihood is proportional to $e^{\beta Y_1}(1+e^{\beta }{)}^{-m}$, also the profile likelihood after maximizing over the unrestricted positive baseline rate. Unequal exposures replace the binomial success probability by $n_1{e}^{\beta }/(n_0+n_1{e}^{\beta })$.

For conditionally independent paired counts with means ${\lambda }_i$ and ${\lambda }_i{e}^{\beta }$, condition on every pair total $N_i=Y_{i0}+Y_{i1}$. The joint conditional likelihood removes all pair-specific rates, leaving the displayed independent binomial distributions. A zero-total pair contributes no information. With unrestricted fixed baseline rates, this conditional likelihood has the same parameter-dependent part as the ordinary profile likelihood; a specified Gamma–Poisson hierarchical model can retain additional information by imposing structure on those rates.

Consider an approximate conditional likelihood in which a paired total $t_i$ leads to a structural zero with probability $\theta$, and otherwise a binomial distribution with success probability $p$. For an observed zero, the posterior cure weight is $q_i=\theta /[\theta +(1-\theta )(1-p{)}^{t_i}]$; for a positive count it is zero. Differentiating the log-likelihood gives the displayed equations at an interior maximum-likelihood estimator, with the weights evaluated at that estimator. Recomputing the weights and then updating the parameters is an expectation-maximization algorithm. Boundary fits must also be considered. These equations describe a conditional mixture: if cure is instead specified before two Poisson distributions are conditioned on their total, its conditional probability generally depends on the total and on the baseline nuisance rate. A zero observed in a short interval is not by itself proof of permanent biological cure.

For a continuous predictor $x$ and a categorical factor, an additive Poisson regression with log link has $\log {\mu }_j(x)=a_j+bx$. The log-mean curves are parallel and ${\mu }_j(x)/{\mu }_k(x)=e^{a_j-a_k}$ is constant in $x$. Adding a predictor-by-factor interaction gives $\log {\mu }_j(x)=a_j+b_{j}x$, so this ratio becomes $e^{a_j-a_k+(b_j-b_k)x}$. With $J$ factor levels, the common-slope restriction has $J-1$ independent constraints. A likelihood-ratio test compares the two deviances with an asymptotic chi-squared distribution on $J-1$ degrees of freedom. Evidence against common slopes establishes varying effects on the log scale; it does not by itself determine their signs or the ordering of the mean curves.

If $C_i\sim \mathrm{Poisson}(T_i{\lambda }_i)$ counts events over $T_i$ days, then the daily average $C_i/T_i$ has mean ${\lambda }_i$ and variance ${\lambda }_i/T_i$. It is not itself a Poisson variable unless $T_i=1$. Known observation periods allow count modelling with a log-exposure offset; otherwise Quasi-Poisson regression may be used as a working mean-variance model, with uncertainty qualified by the missing exposure information.

For independent samples of $n$ observations from Poisson distributions with positive means $a$ and $b$, the maximum-likelihood estimator of $\nu =\log (b/a)$ is the displayed difference when both totals are positive. Its asymptotic variance is $(a^{-1}+b^{-1})/n$, obtained by the delta method or by inverting the two-parameter Fisher information matrix. A zero group total makes its log-mean estimate a boundary value; finite normal intervals require nonzero totals.

For three distinct times, the columns $1,t,t^2$ are linearly independent by the Vandermonde determinant. Thus a quadratic Poisson regression for the log mean can reproduce any three positive time-group means. A grouped goodness-of-fit test of that mean form has zero residual degrees of freedom. Replicated observations still allow checks of the Poisson distribution and overdispersion within each time group; saturation of the mean does not establish the response distribution.

A categorical covariate can be grouped in an interaction term while retaining its original main effect. This shares selected slopes but keeps separate intercepts. Nested Poisson deviance differences test slope pooling and intercept pooling separately.

In a Poisson regression with $\log \mu ={\beta }_0+{\beta }_{T}T+{\beta }_{B}b+{\beta }_{TB}Tb$, the treatment logarithm of the mean ratio at baseline $b$ is ${\beta }_T+b{\beta }_{TB}$. The main treatment coefficient tests the contrast at $b=0$. An overall no-treatment test requires both ${\beta }_T=0$ and ${\beta }_{TB}=0$, with their joint covariance matrix for a Wald test.

For independent Poisson observations with fitted means ${\hat{\mu }}_i$, the deviance from the saturated model iswith $0\log 0=0$.

Write $y_i=e_i+r_i$. For $|r_i|/e_i$ small, the cell contribution to the Poisson deviance expands asSumming gives the Pearson chi-squared statistic as the quadratic approximation. The approximation can be inaccurate for sparse expected counts or cells with relative residuals near minus one.

The full Poisson deviance is $2{\sum }_i[y_i\log (y_i/e_i)-y_i+e_i]$. An unpenalized log-mean intercept has likelihood score ${\sum }_i(y_i-e_i)$, which vanishes at an interior fit. The linear terms then cancel, giving the displayed expression. Without that intercept score equation, discarding the linear terms is not justified. Use the limiting convention $0\log (0/e_i)=0$.

A Gamma regression with logarithmic link models a positive response with $\mathrm{Var}(Y_i\mid X_i)=\varphi {\mu }_i^2$ and $\log {\mu }_i=x_i^T\beta$.

Quasi-Poisson regression uses the mean model $\log {\mu }_i=x_i^T\beta$ and variance relation $\mathrm{Var}(Y_i)=\varphi {\mu }_i$. It estimates $\beta$ through quasi-likelihood and scales its covariance by the estimated dispersion $\varphi$.

With ${\mu }_i=e^{\beta x_i}$, define $U={\sum }_i{x}_i(Y_i-{\mu }_i)$ and $I={\sum }_i{x}_i^2{\mu }_i$. The estimating equation and a Taylor expansion give $\hat{\beta }-{\beta }_0\approx I({\beta }_0{)}^{-1}U({\beta }_0)$. If responses are independent with correct means and $\mathrm{Var}(Y_i)=\varphi {\mu }_i$, then $EU=0$ and $\mathrm{Var}U=\varphi I$. Consequently the first-order estimator variance is $\varphi /I$, even if the working Poisson distribution is wrong. Regular asymptotics require growing information and control of leverage and the Taylor remainder.

For mean ${\mu }_i(\beta )$ and variance proportional to $V({\mu }_i)$, the quasi-score is the sum of $(Y_i-{\mu }_i)V({\mu }_i{)}^{-1}\partial {\mu }_i/\partial \beta$. Setting it to zero estimates $\beta$ without requiring a full response likelihood.

Negative binomial regression models overdispersed counts with a negative binomial distribution, commonly using a logarithmic link for the mean.

For known size $\theta >0$, the negative binomial distribution is an exponential family with natural parameter $\eta =\log [\mu /(\theta +\mu )]<0$ and cumulant $b_{\theta }(\eta )=-\theta \log (1-e^{\eta })$. A generalized linear model can use a logarithmic link function for its mean, even though that link is not canonical. Jointly estimating the size changes the family and variance function, requiring additional updates beyond a single fixed-family GLM fit; this is what MASS::glm.nb does.

With known size $\theta >0$, the mean-dependent log mass of a negative binomial distribution is $y\log \mu -(y+\theta )\log (\mu +\theta )$. Its derivative is $\theta (y-\mu )/[\mu (\mu +\theta )]$, so the saturated mean is $y$, including its boundary value zero. Subtracting the fitted log mass from that maximum gives the displayed deviance. This formula needs no fitted-intercept cancellation. As $\theta \to \infty$, its second term tends to $-2(y_i-e_i)$ and it becomes the full Poisson deviance.

For fixed exposures $x_i\ge 0$, a Poisson exposure model takes independent observationsIt has mean and variance $\theta x_i$ and is therefore heteroscedastic when the exposures differ.

For an independent count with a Poisson distribution of mean $RT$, the rate estimator $N/T$ has variance $R/T$. Its fractional standard deviation is $1/\sqrt{RT}$, so achieving target fractional precision $ϵ$ requires expected count $RT={ϵ}^{-2}$ and exposure $T=1/({ϵ}^{2}R)$.

For independently acquired settings with rates $R_i$, the required total exposure is ${ϵ}^{-2}{\sum }_i{R}_i^{-1}$. Rare settings dominate because they require the same expected count but collect it more slowly. Simultaneous acquisition is a different design, and backgrounds or correlated systematic errors require a different uncertainty calculation.

A rate ratio compares event counts per unit exposure between two conditions. In a Poisson regression with a log link and binary indicator coefficient $\beta$, it equals $e^{\beta }$. Exponentiating an approximate Wald confidence interval for $\beta$ gives an interval for the rate ratio. A rate ratio is distinct from a relative risk or an odds ratio.

Let independent counts satisfy $Y_i\sim \mathrm{Poi}(\theta w_i)$, with positive known exposures $w_i$, $\theta >0$, and $a_n={\sum }_{i=1}^n{w}_i\uparrow a_{\infty }<\infty$. Their likelihood is proportional to ${\theta }^{S_n}{e}^{-a_n\theta }$, where $S_n={\sum }_{i=1}^n{Y}_i$. If $S_n>0$, the positive maximum-likelihood estimator is $S_n/a_n$; when $S_n=0$, no maximum is attained on the open parameter space, while the closed-space maximizer is zero. This extended estimate converges almost surely to $S_{\infty }/a_{\infty }$, where $S_{\infty }\sim \mathrm{Poi}(\theta a_{\infty })$. Finiteness follows from $\mathbb{E}{S}_{\infty }=\theta a_{\infty }$, and the limiting Poisson distribution follows from the finite-sum laws. In particular, the probability of a zero estimate tends to $e^{-\theta a_{\infty }}>0$, proving failure of statistical consistency. The total Fisher information tends to $a_{\infty }/\theta$: infinitely many observations need not provide infinite information.

For $S_r={\sum }_i{x}_i^r$, unweighted least squares through the origin and maximum likelihood giveBoth estimators are unbiased.

The two variances areSince $S_2^2\le S_1{S}_3$ by Cauchy-Schwarz, maximum likelihood has no larger variance, with equality when all positive exposures are equal.

For testing $H_0:\theta =1$ against a larger alternative, approximate size-$0.05$ upper-tail tests reject whenor, respectively,

Logistic regression models the conditional log odds as an affine function of predictors.

For ordered response categories, ordered thresholds ${\theta }_l$ and a common slope vector define this cumulative logistic regression. Increasing $x^T\beta$ shifts probability toward higher categories. A covariate contrast multiplies all corresponding cumulative odds by the same factor. This model is not a canonical baseline-category multinomial logistic regression, so the latter's design-weighted count margins are not generally sufficient for it.

Complete separation means that a predictor direction $v$ strictly separates the binary outcomes as displayed. In logistic regression, replacing $\beta$ by $\beta +tv$ increases each observed-outcome probability as $t$ increases, so the log-likelihood has no finite maximizer. Weak inequalities with some equalities can similarly produce quasi-complete separation. Finite coefficient estimates and ordinary information-matrix inference therefore require an existence check rather than relying only on the formal score equations.

For nonzero design entries the Bernoulli logistic-regression model has strictly concave log likelihood, since $-{\ell }^{\prime \prime }={\sum }_i{x}_i^2{\mu }_i(1-{\mu }_i)>0$. A unique finite estimate exists exactly when its score is positive at minus infinity and negative at plus infinity. Perfect agreement of binary outcomes with the signs of the design can instead give a supremum approached at an infinite coefficient. If every design entry is zero, the coefficient is unidentifiable. Newton method uses the score divided by this observed information, with damping when needed.

A logistic regression can specify each binary response conditionally on recorded past outcomes and baseline covariates. A chain rule for probabilities gives the path likelihood as the product of the conditional Bernoulli distribution masses. This permits dependence within a person's history without treating their outcomes as unconditionally independent. The model must specify the initial history and which past features enter its conditional mean.

The accumulated number of previous successes may enter the conditional mean in a history-dependent logistic regression. Its coefficient exponentiates to the conditional success odds ratio per previous success. For a specified future path, update this cumulative predictor after each outcome before multiplying the conditional probabilities.

A previous outcome may affect a subsequent outcome after controlling both measured covariates and persistent unobserved heterogeneity. This is true state dependence. An observed lagged-outcome association alone does not establish it, because omitted individual propensities can create apparent persistence.

A regression for a categorical response with more than two possible outcomes, parametrizing log probability ratios relative to a baseline category by linear predictors. It preserves categories that a binary logistic regression would merge.

For ordered response categories, model each displayed conditional probability by a grouped-binomial logistic regression. The multinomial likelihood factors exactly into successive conditional binomial likelihoods, using the remaining category total as each stage's denominator. This is a different regression parameterization from baseline-category logits or cumulative odds. Separate cumulative binary indicators from the same person are dependent and cannot be treated as independent likelihood contributions.

For independent multinomial distributions with fixed stratum totals $N_s$ and baseline logits $\log (p_{sl}/p_{sL})=x_s^T{\beta }_l$, the sufficient statistics are the displayed response-specific design-weighted counts, $l<L$. Expanding the log-likelihood gives ${\sum }_{l<L}{\beta }_l^T{T}_l-{\sum }_s{N}_s\log (1+{\sum }_{l<L}{e}^{x_s^T{\beta }_l})$, plus a data-only term. The Fisher-Neyman factorization theorem therefore proves sufficiency; with an unrestricted identifiable canonical parameter, a likelihood-ratio argument proves minimal sufficiency. Factor indicators give the corresponding response/covariate margins.

A logistic-normal count model takes $Y_i\mid {\theta }_i\sim \mathrm{Bin}(n_i,{\mathrm{logit}}^{-1}{\theta }_i)$ and ${\theta }_i=x_i^T\beta +\lambda Z_i+{\eta }_i$, where $Z$ is a stationary unit-variance autoregressive process of order one with coefficient $a$ and the independent ${\eta }_i$ have normal distribution with variance $v$. The marginal latent variance is ${\lambda }^2+v$, and its off-diagonal covariance is ${\lambda }^2{a}^{|i-j|}$. Random success probabilities produce overdispersion relative to the binomial distribution.

A logistic model maps a real-valued score $\eta$ to the probability $p=(1+e^{-\eta }{)}^{-1}$, equivalently $\log (p/(1-p))=\eta$. Logistic regression takes $\eta$ to be an affine function of the predictors.

For an event of probability $p$, the log odds are the natural logarithm of its odds:

Separation in a binary regression model occurs when an affine score assigns all observations from one class to one side of a threshold and all observations from the other class to the other side.

In a binary logistic regression, write the signed directional margin of observation $i$ as $(2y_i-1)x_i^{T}d$. Quasi-complete separation occurs when a nonzero direction $d$ makes all these margins nonnegative, with some zero and some strictly positive. Along coefficients $\beta +td$, $t\to \infty$, each observation's log-likelihood contribution is nondecreasing, and those with positive margins strictly increase toward their limiting value. Hence the unpenalized fit cannot have a finite maximum in that direction. A group containing only outcome zeros is a simple example: sending its indicator coefficient to $-\infty$ improves that group's likelihood while leaving all other observations unchanged. Zero fitted risk is then a boundary limit, and ordinary finite-coefficient Wald confidence intervals are unavailable.

Complete separation means that every signed training margin is strictly positive for some affine score. In unpenalized logistic regression it causes coefficient norms to diverge and prevents a finite maximum-likelihood estimate.

For binary independent responses, logistic regression sets

At an interior maximum-likelihood fit of a logistic-regression model containing an intercept, the intercept score function isConsequently the sum of the fitted probabilities equals the number of observed successes exactly.

For a signed margin $u=yh(x)$, the logistic loss isIt is convex and has derivative ${\varphi }^{\prime }(u)=-e^{-u}/(1+e^{-u})$.

For a positive semidefinite trace ball ${\mathcal{S}}_s$, the quadratic-form hypothesis class isIts empirical logistic risk is convex as a function of $M$.

L1-penalized logistic regression minimizes empirical logistic loss plus $\lambda \Vert \beta {\Vert }_1$. Its subgradient optimality conditions set each logistic score coordinate equal to $-\lambda z_j$, where $z_j=\mathrm{sgn}({\beta }_j)$ off zero and $z_j\in [-1,1]$ at zero.

For independent grouped counts $Y_i\sim \mathrm{Bin}(n_i,p_i)$, grouped-binomial logistic regression specifiesFitting the proportions $Y_i/n_i$ with binomial weights $n_i$ gives the same likelihood.

For independent $Y_{ij}\sim \mathrm{Bin}(m,p_i)$ with $k$ replicates per group and $\mathrm{logit}(p_i)=\mu +{\alpha }_i$, ${\alpha }_1=0$, let $S_i={\sum }_j{Y}_{ij}$ and $N=km$. The likelihood equations give ${\hat{p}}_i=S_i/N$, $\hat{\mu }=\log (S_1/(N-S_1))$, and ${\hat{\alpha }}_i=\mathrm{logit}({\hat{p}}_i)-\hat{\mu }$ when $0<S_i<N$. Boundary totals give infinite logits. With $0<p_1<1$ fixed and $N\to \infty$, the central limit theorem and delta method giveThe other groups do not increase information about the baseline logit when they have free offsets. For two groups, the information matrix for $(\mu ,{\alpha }_2)$ is $\left(\begin{array}{cc}{w}_1+w_2& w_2\\ w_2& w_2\end{array}\right)$ with $w_i=N{p}_i(1-p_i)$, whose inverse has first diagonal entry $1/w_1$.