Symmetry group ($\mathrm{Aut}(X)$)

A symmetry group collects bijections of an object to itself that preserve a specified structure in both directions, with function composition as the operation. For a geometric object one may require Euclidean distances to be preserved, giving an isometry group; for a graph, one preserves adjacency, giving a graph automorphism group. For an unstructured finite set, every permutation is allowed. The structure to be preserved is part of the definition.

For a square with its usual geometric structure, the symmetry group consists of four rotations and four reflections. These form the dihedral group of group order eight. The existing group action example shows how its eight elements move the vertices of a polygon; changing the object to its edges or a marked configuration changes the group action being studied.^[ref]

Specifying the preserved data makes it possible to test a proposed symmetry exactly. After the preserved data are declared, function composition, the identity map, and inverse maps automatically preserve them too. Symmetry transformations therefore form a group, and questions about equivalent configurations become questions about group orbits, stabilizer subgroups, and group homomorphisms. This supplies reusable mathematics for comparing many different kinds of objects.

A symmetry transformation is a bijection preserving the data that define the object's symmetry group. Its individual group elements are symmetry transformations; the group operation combines them by function composition.

Take the four-point setthe vertices of a polygon of a rectangle whose side lengths are four and two. Three choices of structure give three different symmetry groups on these same points.

As an unstructured set, $R$ has all $4!=24$ permutations as symmetries, forming the symmetric group $S_4$. If Euclidean distance is part of the data, the allowed maps are only the identity, a half-turn rotation, and the reflections across the two coordinate axes. They form a Klein four-group of group order four. To see that nothing is missing, fix the image of one vertex of a polygon: its three other distances, $2$, $4$, and $\sqrt{20}$, are distinct, so they uniquely determine the images of the remaining vertices.

Preserving affine structure permits more. The affine mapcycles all four points of $R$. It sends a horizontal side of length four to a vertical side of length two, so it is not an isometry. Yet it preserves the rectangle's vertex set and all affine combinations among those points.

Let $A=\mathrm{diag}(2,1)$ and let $r(x,y)=(-y,x)$ be the square's quarter-turn rotation. ThenConjugating every square symmetry by $A$ gives eight affine maps preserving $R$. This group isomorphism makes the affine symmetry group a dihedral group, although some of its maps do not preserve the original Euclidean distances. There are no further affine symmetries: an affine map preserves the common midpoint of the diagonals, so it must preserve opposite pairs. The image of one vertex has four choices and the image of an adjacent vertex has two; those choices determine the map through three noncollinear points.

ThusAn arbitrary permutation need not be an affine map. For example, exchanging the two top vertices while fixing the two bottom vertices sends the common diagonal midpoint to two different points, which no single affine map can do. The phrase "the symmetry of these points" becomes unambiguous only after stating which structure matters.

Let a group action of $G$ on a set $X$ be given, and let a function $c:X\to C$ attach a colour label to each point. The colour-preserving subgroup of $G$ isIt is a subgroup: the identity map preserves all labels, compositions of two preserving maps preserve them, and replacing $x$ by $g^{-1}x$ shows that inverse maps preserve them too. Extra object data restrict the allowable transformations; the labels are now part of the object rather than bookkeeping for a drawing.

For a geometric square, take $G=D_4$, of group order eight. If all four vertices have the same colour, all eight maps remain. If two adjacent vertices are red and the others blue, only the identity and one reflection remain. Putting the two red vertices opposite one another instead leaves four maps: the identity, a half-turn rotation, and the two diagonal reflections. Giving all four vertices distinct colours leaves only the identity.

The count alone does not specify the pattern. Both two-red examples have the same number of red and blue vertices, but their stabilizer subgroups differ. For a single red vertex, the orbit-stabilizer theorem gives a group orbit of size four and a stabilizer subgroup of group order two, agreeing with the existing group action illustration. Such counts identify equivalent placements without treating each placement as a new geometric case.