For nonzero design entries the Bernoulli logistic-regression model has strictly concave log likelihood, since . A unique finite estimate exists exactly when its score is positive at minus infinity and negative at plus infinity. Perfect agreement of binary outcomes with the signs of the design can instead give a supremum approached at an infinite coefficient. If every design entry is zero, the coefficient is unidentifiable. Newton method uses the score divided by this observed information, with damping when needed.
L1-penalized logistic regression Created 2026-09-28 Updated 2026-11-05
For independent binary responses in a Bernoulli logistic-regression model, let be the summed log-likelihood function. With , minimize
The subgradient optimality condition is
The sign depends on which gradient is named: the likelihood score function equals after division by , while the gradient of the negative average log-likelihood function equals . At a zero coordinate the absolute normalized score function is at most , explaining exact zero components in the coefficient vector.
For a fixed design matrix this is a convex optimization problem, so the conditions are sufficient as well as necessary. An unpenalized regression intercept has its ordinary zero score function instead of an L1 condition. If software minimizes the summed negative log-likelihood function, the matching penalty is .
The first fit is a Bernoulli logistic-regression model for the probability of a positive response. Its fitted linear predictor is
and its fitted probability is . There are eight coefficients. The null degrees of freedom identify , and the residual degrees of freedom are .
Each slope is a change in the log odds per unit of its covariate, holding the others fixed. Exponentiating gives conditional odds ratios; for example a ten-unit glucose increase multiplies fitted odds by , and a ten-year age increase by . These are changes in odds, not multiplicative changes in probability, and the observational associations are not automatically causal effects. The intercept corresponds to all numerical covariates being zero, generally an extrapolation beyond meaningful subjects.
The standard errors come from the inverse Fisher information matrix of the logistic regression. With binomial dispersion parameter fixed at one, the displayed t ratios are asymptotic normal Wald statistics. In the full fit, glucose and the pedigree measure have clear individual evidence of nonzero slopes, with two-sided and . The body-mass and age terms are borderline, with and ; pregnancy count is weaker, . Blood pressure and skinfold have little individual evidence after adjustment. These labels should not be converted into a claim that the weaker predictors have no effect, or used as automatic model-selection rules.
A deviance residual here is
Its square is the observation's contribution to binomial deviance, so the squared residuals sum to the printed residual deviance. The five-number summaries describe the distribution of these signed residuals. The null deviance belongs to an intercept-only logistic regression; the full fitted deviance is . Four Fisher scoring iterations record convergence of the numerical likelihood fit.
The second model uses only glucose:
Its two coefficients leave residual degrees of freedom. A ten-unit glucose increase multiplies its fitted odds by . This differs from the adjusted multiplier , because the models condition on different covariates; both adjustment and noncollapsibility of the odds ratio can change logistic coefficients. The glucose-only model has deviance , so its maximized log-likelihood is lower than the full model's.
For these fixed-dimensional nested models, analysis of deviance and Wilks theorem give
Thus the additional six predictors collectively improve the glucose-only model, even though several individual Wald tests are not significant. These comparisons assume independent subjects, an appropriate logistic mean structure, full rank, and regular finite parameter estimates without separation.
It would be incorrect to infer that absolute fit is good merely because . Individual Bernoulli deviance need not have a chi-squared calibration: there is just one binary observation per saturated probability, and the saturated dimension increases with sample size. A simple counterexample is an intercept-only model with true probability , for which , not one. For absolute adequacy, inspect residual patterns and predicted-risk calibration, use meaningful grouped comparisons when appropriate, or calibrate a chosen fit statistic by parametric bootstrap. The regular nested-model tests above are not subject to that saturated-model dimensionality problem. Only the displayed summaries and a few example rows are supplied, so patient-level diagnostics or complete numerical refitting cannot be reconstructed from this page alone.
The logit link gives . Independence in the Bernoulli logistic-regression model therefore gives the log-likelihood
Differentiate using :
The maximum-likelihood estimator, if finite, solves the single score equation . If at least one is nonzero, the log likelihood is strictly concave at finite , so any root is the unique maximizer.
Use Newton method, equivalently one-parameter iteratively reweighted fitting:
A damped step or a bracketed score solver is useful if a full step overshoots. The finite maximum-likelihood estimate in a one-parameter logistic model requires the score limit at minus infinity to be positive and its limit at plus infinity to be negative. Perfect sign separation of the observations can instead send the estimate to an infinite coefficient; if every , the coefficient is unidentifiable. These possibilities should be checked before claiming convergence to a finite estimate.
Independence in the Bernoulli logistic-regression model gives the log-likelihood
Writing , the score function and curvature are
Thus a finite maximum-likelihood estimate solves . If at least one , the curvature is strictly negative for finite , so any such solution is the unique maximizer.
Use the Newton method, which here coincides with Fisher scoring because the second derivative does not depend on the observed responses:
Step damping or a bracket for the monotone score provides reliable numerical convergence. If all covariates are zero the parameter is unidentifiable; if the outcomes are separated in the direction of the covariates, the likelihood supremum may occur at or rather than at a finite estimate.
For independent patients, the Bernoulli logistic-regression model is
where with nonfrail patients as the reference level in a regression factor. The fitted linear predictor is
The null model fits one intercept, so its 99 residual degrees of freedom imply 100 patients; the full model's confirms this. Unit weights correspond to individual binary responses, not grouped binomial counts.
The Akaike information criterion estimates expected relative out-of-sample discrepancy, using the maximized log-likelihood plus a complexity penalty:
Here , since the Bernoulli dispersion is fixed at one. The saturated model puts probability one on each observed binary outcome and has log likelihood zero, so the binomial deviance is . Consequently
There is no estimated error-variance parameter to count. The printed name “Aikake” should be read as Akaike.
For word-count vector , the fitted logistic model is
Holding other counts fixed, one additional occurrence of "dollar" multiplies the fitted spam odds by .
The logistic classifier is
The coefficients maximize the Bernoulli logistic-regression model likelihood over the training emails.
The code fits a Bernoulli logistic-regression model. Conditionally on the covariates, the responses are independent with
where the design includes the displayed numerical predictors and indicator columns for factor levels. It maximizes
Akaike information criterion is
where is the number of fitted parameters. Backward stepwise selection starts from the full model, tentatively removes each eligible term, chooses the removal producing the lowest AIC, and repeats while AIC decreases. It balances fit against model size rather than testing every coefficient at a fixed significance threshold.
The commands choose observations and predictors; generate a matrix of independent standard-normal values; generate independent binary random variables with success probability ; and report the observed number of successes, . The first glm call then fits the Bernoulli logistic-regression model
The prediction command returns the fitted response probabilities and sums them.
The likelihood is
At the maximum-likelihood estimator, the intercept component of the score function is
Therefore
exactly, by the fitted-mean balance for logistic regression with an intercept.
The second fit uses probit regression,
where is the standard normal distribution function. Its intercept score is
a weighted residual equation. It does not imply , so exact equality is not expected. The output should nevertheless be close to : the data were generated with constant success probability independently of , so the fitted slopes should be small, the linear predictors should cluster near a common intercept, and the score weights should be nearly constant.