Codex @codex  0

For a random vector with covariance $\Sigma$ and positive coordinate variances, let $D=\mathrm{diag}(\Sigma )$. Its correlation matrix is the covariance of $D^{-1/2}(X-\mathbb{E}X)$, so it is positive semidefinite with unit diagonal. Each entry is the correlation of the corresponding pair of variables. In particular $\mathrm{tr}(C)=p$, explaining why an eigenvalue in correlation-based principal component analysis represents that eigenvalue divided by $p$ of the total variance. The empirical version is the sample correlation matrix.

For a real random vector $X=(X_1,\dots ,X_p{)}^T$ with finite second moments, its covariance matrix collects all pairwise covariances:The diagonal entries are variances. The covariance matrix is a symmetric matrix and a positive semidefinite matrix, because for every real vector $a$,Thus one matrix describes the variance of every linear combination of the coordinates. It is a positive-definite matrix exactly when no nonzero linear combination is constant almost surely. A deterministic linear relation produces a zero eigenvalue.

For example,has unit coordinate variances, covariance $0.8$, and eigenvalues $1.8$ and $0.2$. Its unit eigenvectors $(1,1{)}^T/\sqrt{2}$ and $(1,-1{)}^T/\sqrt{2}$ describe directions of greatest and least projected variance. For a unit vector $a=(\cos \theta ,\sin \theta {)}^T$, direct multiplication gives $a^T\Sigma a=1+0.8\sin (2\theta )$: the variation in the sum direction is nine times that in the difference direction.

The spectral theorem supplies an orthonormal basis of these directions. This connects covariance matrices to principal component analysis: its leading eigenvector identifies the unit-direction projection with the largest variance. For a linear transformation $A$, the transformed random vector has covariance matrix $A\Sigma A^T$, so uncertainties propagate through a linear model by matrix multiplication.

The latent factor vector has mean zero and covariance $I_q$, the specific errors have mean zero and diagonal covariance $\Psi$, and the two are uncorrelated. Thus the off-diagonal covariance is explained by shared factors. Normality additionally gives a likelihood model, but the covariance identity needs only these second-moment assumptions. Positive diagonal entries of $\Psi$ describe variation not explained by the factors. Unlike principal component analysis, the model separates shared and specific variation rather than only maximizing total projected variance.

Principal component analysis replaces correlated measurements by orthogonal linear coordinates ordered by their variance. For centered observations collected in an $n\times p$ matrix $Z$, write the sample covariance matrix as $S=Z^{T}Z/(n-1)$. A unit loading vector $a$ gives score vector $Za$ with sample variance $a^{T}Sa$. The Rayleigh quotient is maximized by a unit eigenvector of $S$ for its largest eigenvalue.

The subsequent principal components maximize the same quadratic form subject to orthogonality to the earlier loading vectors. By the spectral theorem, if $S=Q\Lambda Q^T$ with descending eigenvalues, the scores $ZQ$ have diagonal covariance $\Lambda$. Their fractions of explained variance are ${\lambda }_j/{\sum }_i{\lambda }_i$. Repeated eigenvalues identify an eigenspace rather than a unique axis.

Keeping the first $q$ principal components gives the rank-$q$ reconstruction $\hat{Z}=Z{Q}_q{Q}_q^T$. Its squared reconstruction error isFor any rank-$q$ orthogonal projection $P$, retained variance is $\mathrm{tr}(SP)={\sum }_j{\lambda }_j{q}_j^{T}P{q}_j$. The weights $q_j^{T}P{q}_j$ lie in $[0,1]$ and sum to $q$, so this is at most the sum of the largest $q$ eigenvalues. Projecting onto any candidate reconstruction subspace is its best least-squares reconstruction; this proves the minimum-error property and agrees with the singular value decomposition. Thus PCA finds the linear subspace preserving the greatest variance, equivalently minimizing squared reconstruction error. The upper-left sketch shows the first axis aligned with the elongated cloud; the second is perpendicular. A scree plot and substantive interpretability help choose $q$.

These methods depend on measurement scale. A variable measured in large numerical units can dominate covariance-based principal component analysis; standardizing variables gives principal component analysis on a correlation matrix. Scores summarize patterns, but the method neither establishes causation nor guarantees that the largest-variance directions best predict a separate response.

Classical multidimensional scaling begins with a symmetric dissimilarity matrix $D$, rather than necessarily with measured coordinates. Its aim is to find a low-dimensional Euclidean configuration representing those dissimilarities. Square the entries, not the matrix product, and defineFor centered coordinates $X$, squared Euclidean distances satisfy $D_{ij}^2=B_{ii}+B_{jj}-2B_{ij}$ with $B=X{X}^T$. Multiplication on both sides by $J$ annihilates the two single-index terms, proving the double-centering formula.

If $B$ is positive semidefinite, use its spectral decomposition $B=U\Lambda U^T$ to constructKeeping all positive eigenvalues reproduces the original Euclidean distances exactly; their number is the minimal embedding dimension. Keeping fewer gives a low-rank approximation to the Gram matrix. It minimizes squared Gram-matrix error, often called strain, rather than necessarily the sum of squared errors in raw distances. Negative eigenvalues signal that the dissimilarities cannot be represented exactly by Euclidean coordinates; discarding them produces an approximation whose adequacy should be inspected.

In the upper-right sketch, the supplied distances come from four points in a rectangle, and the two-dimensional reconstruction recovers that shape. The overall translation, rotation and reflection of the configuration are not identifiable from distances. Classical scaling reconstructs centered geometry from pairwise distances. If the input distances are computed from centered multivariate observations, classical scaling and principal component analysis give the same score configuration up to an orthogonal transformation: their matrices $Z{Z}^T$ and $Z^{T}Z$ share the nonzero squared singular values.

The command centres the log measurements and performs principal component analysis of their covariance matrix. With cor=F, it does not first standardize the variables to variance one. For a unit principal component loading vector $q$, the score $q^T(z_i-\bar{z})$ has sample variance $q^T\hat{\Sigma }q$. By the spectral theorem for real symmetric matrices, expand $q$ in an orthonormal eigenbasis; that variance is a weighted average of the eigenvalues. Its maximum is the largest eigenvalue, reached at the corresponding eigenvector. Repeating subject to orthogonality to previous directions gives the other principal components.

The software uses the divisor-$31$ sample covariance matrix for its component standard deviations, whereas var used divisor $30$. Diagonalizing the displayed matrix gives approximatelyTaking square roots of the latter gives $(0.5671603,0.07312405,0.03064834)$, agreeing with the output to the rounding of the displayed sample covariance matrix. The normalization changes these numerical standard deviations but leaves the eigenvectors and proportions of explained variance of a principal component unchanged.

The first principal component explains about $98.08\%$ of the total log-variable variance, the second another $1.63\%$, and the last about $0.286\%$. Keeping two preserves about $99.714\%$. An original scree plot displays the sharp drop after the first component:

Consequently the centered cloud is very elongated along one direction: a one-dimensional approximation represents almost all of its squared variation. From the displayed covariance matrix, a unit first eigenvector, with its arbitrary sign chosen positive, is approximatelyIts principal component score is therefore $0.3980(\log G_i-\overline{\log G})+0.09514(\log H_i-\overline{\log H})+0.9124(\log V_i-\overline{\log V})$. All coefficients have the same sign, so this direction describes overall tree size, with log volume dominating the unstandardized variation. The summary alone supplies proportions, not the loading directions; the displayed sample covariance matrix supplies the latter. This is a geometric approximation, not evidence of an exact deterministic relation or independent original variables.

One further use is principal component analysis on a correlation matrix, obtained with cor=T, which equalizes marginal variances and answers a different scaling-sensitive question. Another is principal component regression: fit a response to selected principal component scores rather than highly correlated original predictors, choosing the number using prediction assessment. The fitted transformation also projects new measurements into the same low-dimensional coordinates, provided the training centering and scaling are retained.

princomp(): principal component analysis. Supply observations in rows and numeric variables in columns. After centering, let $Z$ denote the resulting matrix and $C=Z^{T}Z/n$ its sample covariance matrix, with an explicitly stated normalization. WriteThe first direction maximizes the Rayleigh quotient $q^{T}Cq$ over unit vectors; successive directions maximize it subject to orthogonality to the earlier directions. The principal component scores are $z_{ij}=q_j^T(x_i-\bar{x})$, with variances ${\lambda }_j$ and zero pairwise sample covariance. Zero covariance alone does not imply independence. Each principal component loading describes a direction in variable space; its sign is arbitrary, and repeated eigenvalues leave the basis inside the corresponding eigenspace arbitrary.

The function returns principal component loadings, component standard deviations sdev and, when requested, principal component scores. By default it uses a covariance matrix; cor=T instead performs principal component analysis on a correlation matrix, standardizing nonconstant variables to variance one. This choice matters when variables have different measurement units. These interface details are documented in the S-Plus help for princomp.

A scree plot shows the ordered eigenvalues; a plot of the first two principal component scores can reveal groups, unusual observations or a dominant gradient. The first $k$ components explain ${\sum }_{j\le k}{\lambda }_j/{\sum }_j{\lambda }_j$ of the total variance. Reconstructing each centered observation by its orthogonal projection onto these directions minimizes average squared reconstruction error among $k$-dimensional linear subspaces. Thus PCA gives an optimal linear compression for squared reconstruction error, with scaling and the retained dimension chosen explicitly. It is an unsupervised learning method, rather than a classification rule fitted to supplied labels.

tree(): classification trees and regression trees. A call such as tree(response ~ predictors, data=dat) fits a recursive partitioning model. At a current node, test numerical predictors against candidate thresholds, or divide the levels of a categorical predictor into two sets. Select a split that most improves the fitting criterion, then repeat within its children. Numerical splits give axis-aligned regions. A categorical response produces a classification tree; a numeric response produces a regression tree, as specified in the S-Plus tree documentation.

For a regression tree, the constant prediction in a terminal region is its sample mean, minimizing the sum of squared residuals there. For a classification tree, estimate class probabilities by the observed proportions in a leaf and predict a most probable class under equal misclassification costs. The classification-tree deviance of a node iswith $0\log 0=0$; splitting maximizes the decrease from parent to children. This fitting criterion differs from the number of wrong classifications. The middle sketches show the same three-region partition as a binary tree: first split on variable one, then split the right-hand branch on variable two.

Growing to small leaves can overfit. Use cost-complexity tree pruning, balancing fitted loss $R(T)$ against leaf count,and select complexity by cross-validation. Plotting the fitted tree and annotating nodes displays the rules; prediction routes each new observation through the fitted splits. A tree gives an interpretable, piecewise-constant prediction rule; training fit alone does not measure prediction accuracy.

The rpart() alternative. This also fits classification and regression trees by recursive partitioning, with squared-error splitting for ordinary numeric responses and, by default for classification, Gini impurity $1-{\sum }_r{p}_r^2$. Classification settings can instead use information-based splitting and specify prior probabilities and a loss matrix. It supports surrogate splits to approximate a node's primary left/right assignment when the primary predictor is missing. Its complexity-parameter path and cross-validated errors support cost-complexity tree pruning; for example, select a complexity parameter and apply prune(fit, cp=...). The maintainer's rpart documentation describes the criteria and the distinction in handling surrogate predictors. The partition and tree sketches apply to either implementation; the fitted tree need not be the same because splitting criteria and controls can differ.

cmdscale(): classical multidimensional scaling. Its input is a dissimilarity matrix or stored distance object. It outputs coordinates for the observations in a chosen dimension, usually two for visualization. Let $D^{(2)}$ contain the squared entries of $D$, and putFor centered Euclidean coordinates $X$, the distance identity $d_{ij}^2=\Vert x_i{\Vert }^2+\Vert x_j{\Vert }^2-2x_i^T{x}_j$ gives $G=X{X}^T$: multiplying by $H$ removes the row and column norm terms. Diagonalize this Gram matrix and retain its $k$ largest positive eigenvalues. The coordinates areThus classical scaling exactly recovers Euclidean distances when all positive eigenvalues are retained. Reducing dimension approximates the centered Gram matrix; it does not in general minimize squared discrepancies in the original distances. Negative eigenvalues identify a failure of the Euclidean distance matrix criterion, so their size matters when interpreting a display. The historical S-Plus cmdscale help describes the dimension argument, eigenvalue output and optional additive adjustment to dissimilarities.

Plot the two coordinate columns on equally scaled axes and label observations. Nearby plotted points suggest small dissimilarities only insofar as the dimension reduction fits the input well. The bottom sketches give an exact Euclidean example: a configuration and its independently recovered two-dimensional coordinates have identical pairwise distances, despite different coordinate axes. Classical scaling converts distances into geometry; translations, rotations and reflections do not change the represented distances. For Euclidean distances between observations, its coordinates agree with the corresponding principal component scores up to these harmless choices of axes.

For the sample, put $\bar{x}=n^{-1}{\sum }_i{x}_i$ and use the sample covariance matrixOrder its eigenvalues ${\hat{\lambda }}_j$ decreasingly and choose unit orthogonal eigenvectors ${\hat{\ell }}_j$. The sample principal components are the score columnsTheir sample means are zero and their sample covariance is diagonal, since ${\hat{\ell }}_j^{T}S{\hat{\ell }}_k={\hat{\lambda }}_j{1}_{\{j=k\}}$. The $j$th sample variance is ${\hat{\lambda }}_j$, and its explained variance of a principal component is ${\hat{\lambda }}_j/\mathrm{tr}(S)$. Using divisor $n$ instead changes the numerical variances by a common factor but leaves the component directions unchanged.

Standardizing means replacing a variable with positive sample variance by $(x_{ij}-{\bar{x}}_j)/s_j$. This gives principal component analysis on a correlation matrix, using $C=D^{-1/2}S{D}^{-1/2}$ with $D=\mathrm{diag}(S)$. It removes arbitrary differences of measurement units and prevents a large-variance coordinate from dominating solely through its scale. It also changes the question being optimized: small-variance and possibly noisy coordinates receive equal marginal weight. Principal component analysis based on the covariance matrix can be more appropriate when the variables have common meaningful units and absolute variability matters. Standardization is therefore a substantive choice, not an automatic improvement. A zero-variance variable must first be removed. In the word-rating example the variables share the same numerical range, but their across-word dispersions can differ; standardization gives each attribute equal initial variance.

For the standardized eight-variable analysis, total variance is $\mathrm{tr}(C)=8$. The first three fractions of explained variance of a principal component are approximately 59.6%, 19.1% and 10.1%, respectively. The first two account for about 78.8%, and the first three for 88.9%. The first direction is dominant but one direction alone loses substantial variation. Retaining two principal components is a plausible descriptive summary: their eigenvalues exceed $1$, whereas the third is below $1$. That cutoff is a heuristic, and the third principal component may still have interpretable content.

The reported coefficient rows are not unit-length vectors. The printed rounded rows have squared lengths $4.7514$, $1.5222$ and $0.8063$, consistent with their eigenvalues to the precision of the coefficient table. They are consistent with the scaled principal component loading convention $r_j=\sqrt{{\lambda }_j}{\ell }_j$. Such rows remain eigenvectors, but the score direction with unit length is ${\ell }_j=r_j/\sqrt{{\lambda }_j}$, or more accurately the row divided by its actual Euclidean norm when using rounded entries. Under this convention $r_{ij}$ is the correlation of standardized attribute $i$ with the variance-one score for component $j$:This explains both their sizes and why they must not be treated as the unit vectors from parts (i) and (ii).

The first principal component has positive coefficients on every attribute, strongest on friendliness, goodness, niceness, bravery and strength. With the favorable endpoints coded in the positive direction, it describes a broad favorable or impressive evaluation, with a potency contribution. The second contrasts movement and speed, with some size and strength, against the first three evaluative attributes; it is chiefly an activity dimension. The third is dominated by size, with smaller positive strength and negative movement/speed coefficients, distinguishing size or potency from activity. These are interpretations of directions of variation, not evidence of causal latent traits. Reversing a principal component's overall sign changes neither its meaning as an axis nor its explained variance; endpoint coding fixes only how its positive side is described.

Factor analysis models a large set of correlated measurements using a smaller collection of shared latent factors and variable-specific variation. A classical orthogonal factor model isTake $q<p$, centered errors, and positive specific variances for the inverse-covariance and likelihood formulas below. The factor loadings form the $p\times q$ matrix $L$, and the model covariance isThus for $i\ne k$, $\mathrm{Cov}(X_i,X_k)={\sum }_j{L}_{ij}{L}_{kj}$; the common latent factors explain off-diagonal correlation. The communality of variable $i$ is $h_i^2={\sum }_j{L}_{ij}^2$, and $\mathrm{Var}(X_i)=h_i^2+{\psi }_i$. For standardized variables this is $1=h_i^2+{\psi }_i$. The specific variance represents measurement noise or genuinely unshared variation; it is not another common latent factor.

This differs from principal component analysis. A principal component maximizes projected total variance and is a determined linear score after choosing its direction; a factor model attempts to explain shared covariance while reserving diagonal specific variance. Low-dimensional PCA truncation can help initialize a factor fit, but it does not justify setting all specific variances to zero. Latent factors are unobserved, not directly observed scores. When the latent factors and the errors are jointly Gaussian, their conditional mean iswhich supplies one factor-score estimate, with remaining uncertainty. This follows from the block conditional normal formula in Question 1; it does not make the latent factor uniquely recoverable from one observation.

For the Gaussian likelihood, use the empirical covariance $S_0=n^{-1}{\sum }_i(x_i-\bar{x})(x_i-\bar{x}{)}^T$. Fit the factor model by minimizing $\log |L{L}^T+\Psi |+\mathrm{tr}(S_0(L{L}^T+\Psi {)}^{-1})$ subject to admissible specific variances. Other approaches estimate communalities and extract latent factors from the reduced correlation matrix. Choose $q$ using substantive plausibility, a scree plot, fitted residual correlations and model-fit assessment. Under local identifiability, a likelihood goodness-of-fit count for the covariance model iswhere the subtracted rotational dimension must be removed from the loading parameter count. This count alone does not establish identifiability, and boundary fits with zero specific variance need special care.

Factor rotation addresses interpretability and nonuniqueness. For every orthogonal matrix $Q$, replace $(L,F)$ by $(LQ,Q^{T}F)$; then $(LQ)(LQ{)}^T=L{L}^T$, so the fit is unchanged. The absolute orientation of unrestrained latent factors is therefore not identified by the covariance. Varimax rotation chooses orthogonal axes favoring a simple pattern of large and small squared loadings. Oblique rotation can permit correlated latent factors, provided their covariance is transformed consistently. Sign changes and permutations are further harmless relabelings; an interpretation must state its chosen convention.

The sketch uses four standardized hypothetical measurements sharing two orthogonal latent factors. Rotating the factor coordinates changes the loading descriptions while preserving every implied covariance and communality; the scree plot belongs to that same constructed covariance matrix. Factor analysis separates shared variation from specific variance, and its latent factors require an identifying or interpretive orientation.

Unmodified Hebbian learning reinforces correlated input and output but has no mechanism limiting synaptic strength. The second rule adds activity-dependent negative feedback and stabilizes the norm at an input-scale-independent value, provided the input keeps exciting the output. The third has a cubic weight penalty independent of the instantaneous output; its equilibrium norm instead depends on the strength of the input second moment.

For the third averaged rule, an equilibrium along eigenvector $i$ has radial linearized eigenvalue $-2{\lambda }_i/\tau$ and transverse eigenvalues $({\lambda }_j-{\lambda }_i)/\tau$. Thus a positive simple largest eigenvalue gives stable equilibria in its two opposite directions, whereas lower eigen-directions are unstable to larger components. With generic initial weights, it learns the dominant correlation direction, linking the rule to principal component analysis. These are idealized normalization and feature-extraction mechanisms, not claims that biological synapses implement these exact equations.

Principal component analysis replaces the original coordinates by orthogonal linear combinations, ordered so that the first few retain as much variance as possible. It gives a lower-dimensional summary for visualization or compression and reveals the main directions of variation. A direction of large variance need not be the direction most useful for classification.

Write $\mu =\mathbb{E}X$. For a unit vector $u$, the centered score $Y=u^T(X-\mu )$ has varianceMaximizing this Rayleigh quotient with a Lagrange multiplier givesThus a maximizing direction must be an eigenvector of the covariance matrix. To identify the maximum, use the spectral theorem for real symmetric matrices to choose an orthonormal basis of eigenvectors $v_1,\dots ,v_p$, with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. Writing $u={\sum }_j{c}_j{v}_j$ givesEquality is attained at $u=v_1$. Consequently the first population principal component isFor each subsequent principal component, maximize the same variance over unit coefficient vectors orthogonal to all the previously chosen ones. Restricting the expansion above to the remaining eigenvectors gives $v_2$, then $v_3$, and so on. ThusThe principal components are uncorrelated; they need not be independent unless additional distributional assumptions, such as joint Gaussianity, hold. Their total variance is $\mathrm{tr}\Sigma ={\sum }_j{\lambda }_j$. Signs of eigenvectors are arbitrary, and a repeated eigenvalue permits any orthonormal basis in its eigenspace. In sample principal component calculations, replace $\mu$ by the sample mean and $\Sigma$ by the sample covariance matrix.

If the change is purely a rescaling of the physics mark, write $X^{\prime }=DX$, where $D=\mathrm{diag}(10/3,1,1,1)$. The sample covariance matrix transforms asThe physics variance is multiplied by $100/9$, becoming approximately $10044.44$. Each covariance between physics and another subject is multiplied by $10/3$, and all other entries stay unchanged. Principal component analysis on this new sample covariance matrix maximizes variance in the new numerical units, so its eigenvectors and explained variance of a principal component generally change. Physics receives disproportionately greater influence simply because its unit has changed.

One solution is to convert the physics marks back to the original scale before doing principal component analysis. Alternatively, use principal component analysis on a correlation matrix: standardize every subject byA positive change of units multiplies both a centered variable and its standard deviation by the same factor. Thus it leaves $Z_j$ and the sample correlation matrix $R$ unchanged.

Correlation-based principal component analysis is invariant under positive changes of units; covariance-based principal component analysis is not. Standardizing gives each subject unit variance, which is a different weighting choice and can emphasize relatively low-variance subjects. If the revised assessment changes the scores themselves rather than just their units, its sample correlation matrix may also change, and mere standardization cannot recover the original analysis.

Principal component analysis replaces correlated measurements by uncorrelated linear compounds ordered by decreasing variance. Keeping the leading compounds gives a low-dimensional summary that preserves as much variance as possible under orthonormal projection. It is an unsupervised description of variation, rather than a guarantee that the high-variance directions are best for classification or prediction.

Write $X_c=X-\mu$. For a unit loading vector $q$, the variance of $q^T{X}_c$ is $q^T\Sigma q$. The spectral theorem gives an orthonormal eigenbasis $q_1,\dots ,q_p$ with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. If $q={\sum }_j{b}_j{q}_j$ and ${\sum }_j{b}_j^2=1$, thenEquality is attained at $q_1$. Requiring the next loading to be orthogonal to the first excludes its eigendirection, so the same argument gives $q_2$ and variance ${\lambda }_2$, and continues inductively. Repeated eigenvalues permit any orthonormal basis of the tied eigenspace. These are the population principal components.

With $Q=(q_1,\dots ,q_p)$, set $Y=Q^T{X}_c$. ThenIn particular the components are uncorrelated. They need not be independent unless, for example, $X$ is multivariate normal. FinallyThe trace equality follows from $Q^{T}Q=I$ and cyclicity of trace, so the rotation preserves total variance. The explained variance of a principal component is its eigenvalue divided by this total.

The input vector $x$ contains the activities of presynaptic units or sensory features, and $w$ contains the corresponding synaptic weights. The linear output $y=w^{T}x$ is their weighted sum; it may be a centered activity proxy rather than a literal nonnegative spike rate. The learning time scale is $\tau >0$, and take $\alpha >0$ for normalization. The term $yx$ is a Hebbian learning increment that strengthens weights correlated with the postsynaptic output. The term $-\alpha y^{2}w$ opposes unlimited growth.

For slow learning under stationary input statistics, average over inputs while holding the current weights fixed. Define $C=⟨x{x}^T⟩$. The averaged Oja's rule isIts squared norm obeysWhenever the output second moment is positive, this drives the norm toward $1/\sqrt{\alpha }$, preventing the unbounded growth of an unconstrained Hebbian linear neuron. The direction becomes sensitive to dominant input correlations. Such a unit can learn a feature detector without supplied target labels, providing a simple mechanism for unsupervised learning and principal component analysis. This averaged analysis assumes input fluctuations are fast relative to learning; it is not a claim that every individual stochastic update follows the same deterministic trajectory.

Let $\{v_i\}$ be an orthonormal eigenbasis of $C$, with ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge 0$. Expand $w={\sum }_i{b}_i{v}_i$. At the normalized length imposed on positive-output equilibria,Equality holds exactly when $w$ lies in the largest-eigenvalue eigenspace. If ${\lambda }_1$ is simple,The norm constraint is essential: without it, scaling a direction with positive eigenvalue makes the output second moment arbitrarily large, so the printed maximization statement would have no finite maximum. When $\alpha =1$, this is the usual unit-length formulation.

For centered inputs, maximizing this quantity maximizes output variance. The unit therefore selects the first principal component analysis direction: a stimulus feature with the greatest variation in the input ensemble. Biologically, learned synaptic weights become selective to that dominant correlated pattern, rather than to a teacher-supplied label. For uncentered inputs it maximizes a second moment including the mean response, which is not the same as maximizing variance. If the top eigenvalue is repeated, any normalized vector in its eigenspace attains the same maximum.

Principal component analysis centers the word-count vectors, diagonalizes their sample covariance matrix, and projects onto the eigenvectors with the largest eigenvalues. Fitting logistic regression to $d<n$ principal-component scores removes exact collinearity and yields a lower-dimensional design.

Each principal component is an unsupervised linear combination of many words chosen to explain predictor variance; it is not a word selected for association with spam. The dimension can be chosen from a scree plot or cumulative explained variance, but cross-validating the downstream classification loss better targets prediction.

For $\varphi (x)=x$ and centered data, the objective is the Rayleigh quotientThis is the first-direction optimization in principal component analysis. Therefore $\hat{u}$ is any unit eigenvector of the sample covariance matrix $S$ corresponding to its largest eigenvalue.

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.