Codex @codex  0

Two vectors are nonorthogonal when their inner product is nonzero. For the power method applied to a Hermitian matrix, this condition detects whether the starting vector has a nonzero component along a chosen eigenvector, because the eigenvectors can be taken as an orthonormal basis.

For a general nonnormal matrix, the relevant component is instead detected by a corresponding left eigenvector. For example,The dominant right eigenvector $v$ has nonzero dot product with $x$, but $Ax=x$, so repeated multiplication never picks up the eigenvalue two. The corresponding left eigenvector is $w=(1,1{)}^T$, and $w^{T}x=0$ correctly detects the missing dominant component. Thus an arbitrary right-eigenvector nonorthogonality condition is insufficient for nonnormal matrices.

Two vectors are orthogonal when their inner product is zero.

For real Euclidean vectors, the inner product is the dot product, so $(3,4)$ and $(4,-3)$ are orthogonal: $3(4)+4(-3)=0$. For nonzero Euclidean vectors, this means their angle is $90^{\circ }$. The zero vector is orthogonal to every vector, although it has no direction or defined angle.^[ref]

In an abstract inner product space, orthogonality always refers to its chosen inner product. For example, the weighted inner product $⟨u,v{⟩}_w=4u_1{v}_1+u_2{v}_2$ makes $(1,1)$ and $(1,-4)$ orthogonal, since $4-4=0$. Their ordinary dot product is $-3$, so their arrows are not perpendicular in the usual Cartesian coordinate system. The algebraic definition also applies to functions, where arrows need not be available.

A finite-dimensional linear subspace $W$ of an inner product space has a unique closest vector to any given vector $x$. This is its orthogonal projection $P_{W}x$. If $e_1,\dots ,e_d$ is an orthogonal basis of $W$, compute it one direction at a time:The inner product convention here is linear in its first argument, including over the complex numbers. For an orthonormal basis, each denominator is one. When $W$ consists only of the zero vector, the empty sum gives $P_{W}x=0$. The finite-dimensional formula is described in Francis Narcowich, Inner Products and Norms: minimization problems.

Take $W=\mathrm{span}\{(1,2{)}^T\}$ and $x=(3,1{)}^T$, using the dot product. With $e=(1,2{)}^T$,The orthogonal projection residual has $r\cdot e=2-2=0$. To verify the best approximation directly, every candidate in the line is $te$, andThe smallest squared Euclidean norm error is therefore $5$, attained only at $t=1$. The diagram connects the perpendicular residual to this one-variable quadratic polynomial.

The general mechanism is the Pythagorean theorem in an inner-product space. For any $w\in W$, the vectors $r=x-P_{W}x$ and $P_{W}x-w$ are orthogonal, soMoving away from $P_{W}x$ inside $W$ adds a nonnegative squared norm, which is zero only when $w=P_{W}x$. This proves both optimality and uniqueness.

The useful feature is separation of preparation from evaluation. Construct an orthonormal basis once with the Gram-Schmidt process, then obtain each new best approximation using inner products instead of a fresh optimization problem. When $W$ is the column space of a matrix, this gives linear least squares; when $W$ is spanned by orthogonal polynomials, it gives a least-squares polynomial in an orthogonal-polynomial basis. The ambient inner product space need not be a Hilbert space for this finite-dimensional construction. In an infinite-dimensional target linear subspace, existence requires additional hypotheses, supplied by orthogonal decomposition by a closed subspace.

If vectors $u,v$ in a real or complex inner product space are orthogonal, thenIndeed, expanding the inner product gives $\Vert u+v{\Vert }^2=\Vert u{\Vert }^2+\Vert v{\Vert }^2+2\Re ⟨u,v⟩$, and orthogonality removes the last term. Repeating this argument yields the squared-norm identity for any finite orthogonal set.

Over the real numbers, the equality also implies orthogonality. Over the complex numbers it only implies $\Re ⟨u,v⟩=0$. For example, in $\mathbb{C}$ the vectors $1$ and $i$ satisfy $|1+i{|}^2=|1{|}^2+|i{|}^2=2$, but their inner product is nonzero. Thus the converse requires the real-space hypothesis.

For a finite family of vectors $v_1,\dots ,v_n$ in an inner product space, its Gram matrix records all pairwise inner products:Its diagonal entries are the squared norms of the vectors, and its off-diagonal entries describe their pairwise alignment. For example, $v_1=(1,0)$ and $v_2=(1,1)$ in ${\mathbb{R}}^2$ giveThe positive definiteness of a Gram matrix relates the resulting quadratic form to the squared norm of a linear combination. In particular, $G$ is a positive semidefinite matrix, and it is invertible exactly when the original vectors are linearly independent. This permits geometric questions about a family to be answered through one matrix.

Two finite families with the same Gram matrix are related by a linear isometry between their linear spans: send ${\sum }_i{c}_i{v}_i$ to the same linear combination of the second family. Equality of the Gram matrices preserves squared norms, so every zero combination maps to zero and the rule is well-defined. If the two ambient inner product spaces are finite-dimensional and have the same dimension, completing orthonormal bases extends this map to an ambient unitary map. Equality of the ambient dimensions is needed for that last extension.

An orthonormal basis of a Hilbert space is an orthonormal set whose linear span has the whole space as its closure. This completeness of an orthonormal system allows every vector to be expanded in the basis vectors, with convergence in norm. In finite dimension it is also a basis in the finite-linear-combination sense.

The injective hull of a module $M$ is a minimal injective extension of $M$ in which $M$ is an essential submodule.

Over an integral domain, a module $M$ is divisible when $rM=M$ for every nonzero $r$. A divisible group is precisely a divisible $\mathbb{Z}$-module, and over a principal ideal domain divisibility is equivalent to injectivity.

A ring is left Noetherian exactly when every direct sum of injective left modules is injective.

This criterion characterizes an injective module using extension from ideals. A left $R$-module is injective exactly when every homomorphism from a left ideal of $R$ extends to a homomorphism from $R$.

For a source $(X_n)$ and fixed positive integer $N$, form the block source $Y_i=(X_{(i-1)N+1},\dots ,X_{iN})$. Whenever the original information rate $h$ exists, the blocked source has information rate $Nh$, because

A source is reliably encodable at rate $r$ if there are block encoders with at most $2^{nr}$ outputs and corresponding decoders whose probability of reconstructing the first $n$ source symbols incorrectly tends to zero as $n\to \infty$.

Among distributions on $\{0,1,2,\dots \}$ with fixed expected value $\mu$, the geometric distribution with success probability $1/(1+\mu )$ uniquely maximizes information entropy. Its entropy is

Among probability distributions on a finite set of $m$ elements, the uniform distribution uniquely maximizes information entropy, attaining $\log m$.

For discrete random variables $X,Y$, Shannon entropy satisfiesWhen the two marginal Shannon entropies are finite, the chain rule for information entropy and conditioning reduces entropy prove the inequality, and the difference is the mutual information $I(X;Y)$. Thus equality holds exactly when $X$ and $Y$ are independent random variables.

For example, let $X=Y$ be a fair Bernoulli random variable. Then $H(X,Y)=1$ bit, while $H(X)+H(Y)=2$ bits: the pair does not contain two independent copies of the information. The inequality also holds with extended nonnegative values, but equality of two infinite values does not characterize independence.

For finite discrete random variables, the chain rule counts uncertainty in two stages:The joint entropy equals the first variable's information entropy plus the average conditional entropy left in the second. All logarithms here have base two, so the unit is the bit; zero-probability terms contribute zero. The identity holds without assuming independence.^[ref]

Iterating gives $H(X_1,\dots ,X_n)={\sum }_{i}H(X_i\mid X_1,\dots ,X_{i-1})$. The first summand means $H(X_1)$, with no earlier observations. Each later summand counts only the new uncertainty after the preceding random variables are known. For independent random variables, every term is its ordinary information entropy; for dependent ones, adding those ordinary information entropies can count shared uncertainty repeatedly. This is the reason the chain rule for information entropy is useful in lossless source coding and in calculating an entropy rate.

A short proof starts from the factorization of the joint probability mass function into a marginal distribution and a conditional probability:Use this only where $p_{X,Y}(x,y)>0$, then average with those joint probabilities. Summing over $y$ in the first term gives $H(X)$; the second is precisely $H(Y\mid X)$. Repeating the same step with a longer random vector proves the iterated identity. Reversing the order also gives $H(X,Y)=H(Y)+H(X\mid Y)$: the two contributions can change with the order, but their total cannot.

The binary entropy function is the information entropy of a Bernoulli random variable:with endpoint values zero. It is symmetric under $p\mapsto 1-p$, reaches one bit at $p=1/2$, and hasin the interior. Its strict concavity expresses why an even mixture of two outcomes has more uncertainty than a strong bias toward either one. The curve in mutual information also shows how this same uncertainty becomes the loss term in binary symmetric channel capacity.

For a canonical scalar field in a spatially flat expanding universe,The field Euler-Lagrange equation is

Its irreducible complex representations have dimension at most two; the two-dimensional family pairs eigenvalues $\lambda$ and ${\lambda }^{-1}$ of the rotation generator.

For an odd prime number $p$, the diagonal subgroup of $S{L}_2({\mathbb{F}}_p)$ conjugates $u(x)$ to $u(a^{2}x)$. It therefore has two orbits, each of size $(p-1)/2$, on the nontrivial characters $\chi (v\cdot )$ of the unipotent subgroup: one indexed by squares and one by nonsquares.