Matrix

The set of all invertible matrices forms a group: the general linear group with matrix multiplication. Non-invertible matrices don't form a group due to the lack of inverse.

Transpose of a matrix multiplication

When it distributes it inverts the order of the matrix multiplication:

(M N)^{T} = N^{T} M^{T}

Matrix multiplication

Since a matrix

M

can be seen as a linear map

f_{M} (x)

, the product of two matrices

M N

can be seen as the composition of two linear maps:

f_{M} (f_{N} (x))

One cool thing about linear functions is that we can easily pre-calculate this product only once to obtain a new matrix, and so we don't have to do both multiplications separately each time.

Application of systems of linear equations

No 2x2 examples please. I'm talking about large matrices that would be used in supercomputers.

System of linear equations algorithm

Conjugate gradient method

For positive definite matrices only.

TODO application.

TODO speedup over algorithm for general matrices.

www.studentclustercompetition.us/ comments:

The HPCG benchmark uses a preconditioned conjugate gradient (PCG) algorithm to measure the performance of HPC platforms with respect to frequently observed but challenging patterns of computing, communication, and memory access. While HPL provides an optimistic performance target for applications, HPCG can be considered as a lower bound on performance. Many of the top 500 supercomputers also provide their HPCG performance as a reference.

Application of matrix multiplication

math.stackexchange.com/questions/41706/practical-uses-of-matrix-multiplication/4647422#4647422 highlights deep learning applications.

Matrix multiplication algorithm

math.stackexchange.com/questions/30330/fast-algorithm-for-solving-system-of-linear-equations/259372#259372

General Matrix Matrix Multipliation (GEMM)

The terminology GEMM is present on BLAS, and has stuck pretty much.

Applications of eigenvalues and eigenvectors

Eigenvalue

See: eigenvalues and eigenvectors.

Spectrum (functional analysis)

Set of eigenvalues of a linear operator.

Continuous spectrum (functional analysis)

Unlike the simple case of a matrix, in infinite dimensional vector spaces, the spectrum may be continuous.

The quintessential example of that is the spectrum of the position operator in quantum mechanics, in which any real number is a possible eigenvalue, since the particle may be found in any position. The associated eigenvectors are the corresponding Dirac delta functions.

Eigendecomposition of a matrix

Every invertible matrix

M

can be written as:

M = Q D Q^{- 1}

where:

$D$ is a diagonal matrix containing the eigenvalues of $M$
columns of $Q$ are eigenvectors of $M$

Note therefore that this decomposition is unique up to swapping the order of eigenvectors. We could fix a canonical form by sorting eigenvectors from smallest to largest in the case of a real number.

Intuitively, Note that this is just the change of basis formula, and so:

$Q^{- 1}$ changes basis to align to the eigenvectors
$D$ multiplies eigenvectors simply by eigenvalues
$Q$ changes back to the original basis

Eigendecomposition of a real symmetric matrix

The general result from eigendecomposition of a matrix:

M = Q D Q^{- 1}

becomes:

M = O D O^{T}

(2)

where

O

is an orthogonal matrix, and therefore has

O^{- 1} = O^{T}

Sylvester's law of inertia

The main interest of this theorem is in classifying the indefinite orthogonal groups, which in turn is fundamental because the Lorentz group is an indefinite orthogonal groups, see: all indefinite orthogonal groups of matrices of equal metric signature are isomorphic.

It also tells us that a change of basis does not the alter the metric signature of a bilinear form, see matrix congruence can be seen as the change of basis of a bilinear form.

The theorem states that the number of 0, 1 and -1 in the metric signature is the same for two symmetric matrices that are congruent matrices.

For example, consider:

A = [2223]

The eigenvalues of

A

are

1

and

4

, and the associated eigenvectors are:

v_{1} = [- 2, 1]^{T} v_{4} = [2 / 2, 1]^{T}

(2)

symPy code:

A = Matrix([[2, sqrt(2)], [sqrt(2), 3]])
A.eigenvects()

and from the eigendecomposition of a real symmetric matrix we know that:

A = P D P^{T} = [- 2 1 2 / 2 1] [1004] [- 2 2 / 2 11]

(3)

Now, instead of

P

, we could use

P E

, where

E

is an arbitrary diagonal matrix of type:

[e_{1} 0 0 e_{2}]

(4)

With this, would reach a new matrix

B

B = (P E) D (P E)^{T} = P (E D E^{T}) P^{T} = P (E E D) P^{T}

(5)

Therefore, with this congruence, we are able to multiply the eigenvalues of

A

by any positive number

e_{1}^{2}

and

e_{2}^{2}

. Since we are multiplying by two arbitrary positive numbers, we cannot change the signs of the original eigenvalues, and so the metric signature is maintained, but respecting that any value can be reached.

Note that the matrix congruence relation looks a bit like the eigendecomposition of a matrix:

D = S M S^{T}

(6)

but note that

D

does not have to contain eigenvalues, unlike the eigendecomposition of a matrix. This is because here

S

is not fixed to having eigenvectors in its columns.

But because the matrix is symmetric however, we could always choose

S

to actually diagonalize as mentioned at eigendecomposition of a real symmetric matrix. Therefore, the metric signature can be seen directly from eigenvalues.

Also, because

D

is a diagonal matrix, and thus symmetric, it must be that:

S^{T} = S^{- 1}

(7)

What this does represent, is a general change of basis that maintains the matrix a symmetric matrix.

Congruent matrix

Two symmetric matrices

A

and

B

are defined to be congruent if there exists an

S

G L (n)

such that:

A = S B S^{T}

Matrix congruence can be seen as the change of basis of a bilinear form

From effect of a change of basis on the matrix of a bilinear form, remember that a change of basis

C

modifies the matrix representation of a bilinear form as:

C^{T} M C

So, by taking

S = C^{T}

, we understand that two matrices being congruent means that they can both correspond to the same bilinear form in different bases.

Matrix similarity

Metric signature

Metric signature matrix

Eigenvector

See: eigenvalues and eigenvectors.

Eigenvectors and eigenvalues of the identity matrix

math.stackexchange.com/questions/1507290/linear-algebra-identity-matrix-and-its-relation-to-eigenvalues-and-eigenvectors/3934023#3934023

Spectral theorem

Hermitian matrix (complex analogue of symmetric matrix)

Hermitian operator

This is the possibly infinite dimensional version of a Hermitian matrix, since linear operators are the possibly infinite dimensional version of matrices.

There's a catch though: now we don't have explicit matrix indices here however in general, the generalized definition is shown at: en.wikipedia.org/w/index.php?title=Hermitian_adjoint&oldid=1032475701#Definition_for_bounded_operators_between_Hilbert_spaces

Riesz representation theorem

Kronecker product

Named matrix

Dense and sparse matrices