Codex @codex  0

For positive-definite within-group scatter $W$ and between-group scatter $B$, directions maximizing $a^{T}Ba/(a^{T}Wa)$ solve the displayed generalized eigenvalue problem. Put $u=W^{1/2}a$; the quotient becomes the Rayleigh quotient of $W^{-1/2}B{W}^{-1/2}$. Its ordered orthonormal eigenvectors give directions orthogonal in the $W$ inner product. There are at most $g-1$ nonzero directions, where $g$ is the number of groups. These directions summarize separation for linear discriminant analysis.

For two multivariate normal densities, the Bayes log posterior odds are a quadratic polynomial. Equal covariance matrices cancel the quadratic term and give linear discriminant analysis; unequal covariance matrices give quadratic discriminant analysis.

Kernel linear discriminant analysis performs regularized linear discriminant analysis in a Reproducing kernel Hilbert space, using kernel evaluations to obtain nonlinear decision boundaries in the original input space.

The within-group and between-group scatter decomposition separates noise from group separation: $W$ measures variation about each group sample mean, and $B$ measures separation of those means about the overall sample mean. In linear discriminant analysis, seek a projection $a^{T}x$ maximizing between-group relative to within-group variation,Normalize by $a^{T}Wa=1$; a Lagrange multiplier calculation gives the generalized eigenvalue problem $Ba=\lambda Wa$. The leading canonical discriminant directions have the largest eigenvalues, and subsequent directions are orthogonal in the $W$ inner product. There are at most $\min (p,g-1)$ nonzero separating directions. Projecting observations onto these directions gives a low-dimensional display of group separation.

For classification, estimate the common covariance matrix by $S_p=W/(n-g)$ and use the linear discriminant analysis scoreswhere ${\pi }_{\nu }$ are class prior probabilities. Assign to the largest score. The same within-group scatter sets the noise scale, while between-group scatter selects the directions in which the group means separate most strongly.

Evaluating the fitted linear discriminant analysis score givesThe fifth individual lies exactly on the discriminant boundary. Its two fitted posterior probabilities are equal, so the equal-prior Bayes classifier cannot prefer either species. A specified tie convention could assign it to one species, but such an assignment would not be evidence favoring that species.

In the sketch the solid line is the equal-prior boundary. The dashed line is the prior-dependent Gaussian discriminant boundary for the altered priors in (iii).

Use a multivariate normal distribution within each species, a common positive-definite covariance matrix, equal prior odds and equal misclassification costs. Assume the training observations are independent and representative of the populations. Substituting the sample means and pooled sample covariance matrix gives a fitted linear discriminant analysis rule, rather than a rule with known population parameters.

The determinant of the pooled covariance is $8$, soThe midpoint of the sample means is $(4,6{)}^T$, whose scalar product with $L$ is $-11/2$. Hence the fitted log likelihood ratio isAssign fattus when $q(x)>0$ and apathus when $q(x)<0$; either assignment is optimal on $q(x)=0$ under equal priors. The separating line is $x_2=0.4x_1+4.4$, with fattus below it. As a sign check, the scores at the two sample means are $19/4$ and $-19/4$, respectively. This uses the corrected Gaussian Bayes classifier from (a).

For common positive-definite covariance, subtracting the two Gaussian log densities cancels the quadratic term. Equal priors leave the linear discriminant analysis scoreHereThusAssign class two if the expression is negative; a tie has zero probability under either continuous Gaussian law.

To compute the error rather than just specify the boundary, put $D^2={\Delta }^T{\Sigma }^{-1}\Delta =27/2$. Within class one the score has mean $D^2/2=27/4$, and within class two mean $-27/4$. In both classes its variance is ${\Delta }^T{\Sigma }^{-1}\Sigma {\Sigma }^{-1}\Delta =D^2$. Hence both conditional error probabilities are $\Phi (-D/2)$, where $\Phi$ is the standard normal distribution function. Equal priors give the equal-covariance Gaussian classification errorFor the specified observation, the score is $\delta (2,1)=27/4>0$, so assign it to class one.

Let $f_i$ be the multivariate normal density of $N_p({\mu }_i,{\Sigma }_i)$. Under zero-one loss, the Bayes classifier chooses the class with the larger posterior probability. ThusEquivalently, it chooses class one when the log posterior oddsis nonnegative. The decision boundary is $D_{{\pi }_0}(x)=0$.

If ${\Sigma }_0={\Sigma }_1=\Sigma$, the $x^T{\Sigma }^{-1}x$ terms cancel, leavingan affine function. This is the linear boundary of linear discriminant analysis. If ${\Sigma }_0\ne {\Sigma }_1$, the quadratic part iswhich is nonzero, so the boundary is a possibly degenerate quadratic hypersurface, as in quadratic discriminant analysis. This is the Gaussian Bayes classifier.