Solution (source code)

= Solution

The <Riesz-Markov-Kakutani representation theorem> identifies the dual of the complex <space of continuous functions on a compact space> $C(K)$ with finite regular complex <Borel measures> on $K$: each <bounded linear functional> has a unique representation
$$
\boxed{L(f)=\int_K f\,d\mu,\qquad\|L\|=|\mu|(K).}
$$
Here $|\mu|$ is the <variation measure>; positive functionals correspond exactly to positive regular measures. This is the measure representation theorem, rather than the Hilbert-space representation of a functional by a vector.

For the spectral construction use the convention that the Hilbert <inner product> is linear in its first argument. A unital subalgebra of $\mathcal B(H)$ is understood to contain $I_H$: an algebra whose abstract identity is a smaller projection could not give a resolution normalized by $P(K)=I_H$. On a <compact Hausdorff space>, a resolution of the identity is understood to be a normalized regular <projection-valued measure>: its scalar measures are regular, its values are <orthogonal projections>, and it is countably additive in the strong operator topology. Regularity is part of the usual convention needed for the uniqueness assertion; the statement is interpreted in this sense.

The <Commutative Gelfand--Naimark theorem> makes the Gelfand transform an isometric unital star-isomorphism $A\to C(K)$, where $K=\Phi_A$ is compact Hausdorff. Denote its inverse by $\pi:C(K)\to A\subset\mathcal B(H)$. For $x,y\in H$, the <bounded linear functional> $f\mapsto\langle\pi(f)x,y\rangle$ has <norm> at most $\|x\|\|y\|$. Riesz-Markov-Kakutani gives a regular complex measure $\mu_{x,y}$ such that
$$
\langle\pi(f)x,y\rangle=\int_K f\,d\mu_{x,y},\qquad
|\mu_{x,y}|(K)\le\|x\|\|y\|.
$$
Uniqueness makes these measures sesquilinear in $x,y$. For $f\ge0$, the identity $\pi(f)=\pi(\sqrt f)\pi(\sqrt f)^*$ shows positivity, so $\mu_{x,x}$ is positive with total mass $\|x\|^2$.

For every Borel set $B$, the bounded sesquilinear form $(x,y)\mapsto\mu_{x,y}(B)$ determines an operator $P(B)$ by Hilbert-space <Riesz representation theorem>, with
$$
\langle P(B)x,y\rangle=\mu_{x,y}(B).
$$
In particular $0\le P(B)\le I$ and $P(B)$ is self-adjoint. We now verify the projection identity rather than assume it.

For continuous $f$, uniqueness of the representing measure applied to continuous test functions gives
$$
\mu_{\pi(f)x,y}=f\mu_{x,y},\qquad
\mu_{x,\pi(\overline f)y}=f\mu_{x,y}.
$$
These identities imply $P(B)\pi(f)=\pi(f)P(B)$. Testing once more against a continuous $g$ gives
$$
\langle\pi(g)P(B)x,y\rangle=\int_B g\,d\mu_{x,y}.
$$
The restriction $1_B\mu_{x,y}$ of a finite regular Borel measure is still regular, so uniqueness in Riesz-Markov-Kakutani yields $\mu_{P(B)x,y}=1_B\mu_{x,y}$. Consequently, for any Borel $C$,
$$
\langle P(C)P(B)x,y\rangle=\mu_{P(B)x,y}(C)
=\mu_{x,y}(B\cap C),\qquad
\boxed{P(C)P(B)=P(B\cap C).}
$$
Taking $C=B$ proves that $P(B)$ is an <orthogonal projection>. Also $P(K)=I$ and $P(\varnothing)=0$. For disjoint $B_j$, these projections are orthogonal, and scalar countable additivity gives weak countable additivity. If $B=\bigcup_j B_j$, then $R_N=P(B)-\sum_{j\le N}P(B_j)$ is the projection of the remaining union and
$$
\|R_Nx\|^2=\mu_{x,x}\left(\bigcup_{j>N}B_j\right)\longrightarrow0.
$$
Thus countable additivity holds in the <strong operator topology>. The scalar measures are the regular measures already constructed. This completes the <scalar-measure construction of a projection-valued measure>.

By the integral theorem permitted in the question, continuous $f$ satisfies $\langle(\int f\,dP)x,y\rangle=\int f\,d\mu_{x,y}=\langle\pi(f)x,y\rangle$, and hence $\int f\,dP=\pi(f)$. For $T\in A$, this proves the <spectral theorem for a commutative operator algebra>:
$$
\boxed{T=\int_K\widehat T\,dP.}
$$
Any other regular resolution giving these integrals has the same scalar integrals on all of $C(K)$; Riesz-Markov-Kakutani uniqueness forces the same scalar measures and therefore the same projections on every Borel set.

For nonempty open $U\subset K$, compact Hausdorff normality supplies a nonzero <continuous function> $f$ supported in $U$. If $P(U)=0$, the stated squared-norm identity for spectral integrals gives $\pi(f)=\int f\,dP=0$, contradicting the <isometry> of $\pi$. This proves the <full support of a faithful spectral measure> property
$$
\boxed{P(U)\ne0\quad\text{for every nonempty open }U\subset K.}
$$
Faithfulness of the representation is essential here.

The <spectral theorem for normal operators> says that a bounded <normal operator> $T$ on a nonzero complex <Hilbert space> has a unique regular projection-valued measure $E$ on $\sigma(T)$ such that $T=\int z\,dE(z)$. Its support is all of $\sigma(T)$, and bounded Borel functions have the associated <Borel functional calculus for a normal operator>.

For the proof sketch, $A=C^*(I,T)$ is commutative because $T$ commutes with $T^*$; polynomials in these two operators commute, as do their <norm> limits. The map $\Phi_A\to\sigma(T)$, $\varphi\mapsto\varphi(T)$, is onto by the <character of an algebra> formula and <spectral permanence for C-star algebras>. It is one-to-one because a <character of an algebra> preserves the star operation and its values on $T,T^*$ determine it on their dense polynomial algebra. It is therefore a <homeomorphism> from compact $\Phi_A$ to the Hausdorff spectrum. Transport the resolution just constructed through this <homeomorphism>. It gives the formula for $T$ and full support; conversely a regular resolution for $T$ gives the same integrals for polynomials in $T,T^*$, hence by density the same continuous functional calculus and the same resolution. This argument works without separability of $H$.

Finally choose disjoint nonempty relatively open sets $U,V\subset\sigma(T)$ around two distinct spectral points, and put $Q=E(U)$. Full support gives $Q\ne0$ and $E(V)\ne0$, while $QE(V)=0$, so $Q\ne I$. The spectral integral, or multiplicativity of its Borel calculus with $1_U$, gives $QT=TQ$ and also $QT^*=T^*Q$. Thus $Y=QH$ is closed, nonzero and proper, and $T(Qx)=Q(Tx)\in Y$. The <spectral projection gives a reducing subspace> conclusion is
$$
\boxed{Y=E(U)H\text{ is a nontrivial closed invariant subspace of }T.}
$$
In fact it is a reducing subspace, since it is also invariant under $T^*$.