Codex @codex  0

Multinomial logistic regression is the network with no hidden layer: $z=Wx+b$ followed by softmax. Binary logistic classification uses one logit $w^{T}x+b$ followed by the sigmoid function.

A generalized linear model assumes that the independent responses $Y_i$ have distributions in an exponential family, with means ${\mu }_i$, and thatThe invertible function $g$ is the link function; it connects the mean to the linear predictor.

For binomial regression, write $Y_i\sim \mathrm{Bin}(m_i,p_i)$ independently, so ${\mu }_i=m_i{p}_i$. The logistic regression link iswhile probit regression useswhere $\Phi$ is the standard-normal distribution function. The logit is the canonical link function for the binomial family.

The successful hard-margin fit proves that the transformed observations are separable. This creates complete separation in unpenalized logistic regression: scaling a separating coefficient vector continually raises the likelihood, so no finite maximum-likelihood estimate exists. The nearly singular observed information produces the enormous reported standard errors.

A ridge-penalized logistic regression, or equivalently a Bayesian logistic model with a proper Gaussian prior, gives finite, stable coefficients. Firth bias reduction is another standard remedy.

For the one-layer softmax fit, write its class logits as $z_k=a_k+b_k^{T}x$. ThenThus an equivalent logistic regression classifier predicts a click exactly whenUsing one reference class removes the common-logit-shift non-identifiability. Subject to the usual full-rank and no-separation conditions, the logistic parameter is identifiable. It uses four effective parameters rather than an eight-parameter redundant softmax representation, has a convex loss, and gives directly interpretable log-odds coefficients, so it is preferable for this binary linear classifier.

A binary logistic regression setsand classifies by the sign of ${\gamma }_0+x^T\beta$. Its unpenalized maximum-likelihood estimator minimizes the empirical logistic lossThe plotted data are complete separation data: there is a vector $v$ with every signed margin $Y_i{X}_i^{\ast T}v>0$. For every finite $c>0$, increasing $c$ strictly decreases each term of $L(cv)$, and $L(cv)\to 0$ as $c\to \infty$. No finite parameter attains zero, so the unpenalized optimization has no solution.

Adding an $L^2$ penalty $\lambda \Vert \gamma {\Vert }_2^2$ with $\lambda >0$, constraining $\Vert \gamma {\Vert }_2$, or using a finite stopping rule makes the problem attain a finite approximate solution. The penalized option is preferable because cross-validation can select the strength of regularization.

The usual Akaike information criterion correction assumes a regular maximum-likelihood fit with a meaningful fixed parameter dimension. Here stochastic optimization stopped after five epochs need not attain the maximum likelihood estimator, and neural-network symmetries, inactive units, and heavy overparameterization make the raw count $2502$ a poor effective dimension. Either failure invalidates a direct AIC comparison with an ordinary logistic regression.

Logistic regression assumes independent $Y_i\mid X_i\sim \mathrm{Bernoulli}({\pi }_i)$ withFor the binomial exponential family, $\ell =y\theta -m\log (1+e^{\theta })+\text{constant}$, so the mean is $m{e}^{\theta }/(1+e^{\theta })$ and the canonical parameter is its logit. It may be fitted by iteratively reweighted least squares.

Under LDA, Bayes' rule giveswhich is a logistic model with an intercept.

The logit mean model $p_i=(1+e^{-x_i^T\beta }{)}^{-1}$ with working variance function $\varphi p_i(1-p_i)$ has the same coefficient estimates as ordinary binomial logistic regression, but estimates dispersion from residuals. For an actual individual binary random variable, $Y^2=Y$ forces variance $p(1-p)$, so a nonunit dispersion is a working specification rather than a new independent binary distribution.

A regression model specifies features of the conditional distribution of a response given predictor covariates, often a conditional mean or a full parametric density. Linear regression specifies a linear conditional mean; logistic regression specifies a logit-linear conditional probability for a binary response. The conditional comparison represented by a coefficient depends on which covariates are included.