The quadratic Yang-Mills operator has zero directions along gauge orbits, so it has no propagator until one chooses a gauge fixing. The Faddeev-Popov determinant generated by this choice is represented by anticommuting Faddeev-Popov ghost fields, which cancel unphysical gauge-field contributions in loop calculations. A Nakanishi-Lautrup field imposes the gauge condition algebraically and lets the gauge-fixing plus ghost action be written as a BRST-exact term with off-shell nilpotency.
For the convention in the question, a finite Yang-Mills gauge transformation acts covariantly on the field strength:
or with and exchanged if the opposite convention is used for . Cyclicity of the matrix trace gives
so the Yang-Mills theory Lagrangian is gauge invariant.
The quadratic gauge-field operator has zero directions along each gauge orbit. It therefore has no inverse on the full field space. Gauge fixing removes this degeneracy and produces a propagator, while the Faddeev-Popov determinant accounts for the corresponding Jacobian.