Intuitive definition: real group of rotations + reflections.
Mathematical definition that most directly represents this: the orthogonal group is the group of all matrices that preserve the dot product.
When viewed as matrices, it is the group of all matrices that preserve the dot product, i.e.:
This implies that it also preserves important geometric notions such as norm (intuitively: distance between two points) and angles.
This is perhaps the best "default definition".
We looking at the definition the orthogonal group is the group of all matrices that preserve the dot product, we notice that the dot product is one example of positive definite symmetric bilinear form, which in turn can also be represented by a matrix as shown at: Section "Matrix representation of a symmetric bilinear form".
By looking at this more general point of view, we could ask ourselves what happens to the group if instead of the dot product we took a more general bilinear form, e.g.:
The answers to those questions are given by the Sylvester's law of inertia at Section "All indefinite orthogonal groups of matrices of equal metric signature are isomorphic".
Note that:
and for that to be true for all possible and then we must have:
i.e. the matrix inverse is equal to the transpose.
Conversely, if:
is true, then
These matricese are called the orthogonal matrices.
TODO is there any more intuitive way to think about this?
Or equivalently, the set of rows is orthonormal, and so is the set of columns. TODO proof that it is equivalent to the orthogonal group is the group of all matrices that preserve the dot product.