Solution (source code)

= Solution

At <equilibrium>, the two horizontal equations imply $\hat X_2=\hat X_1$ and then $k_1\hat X_1=0$, so both vanish. Adding the two vertical equations gives $k_1\hat Y_1=-(m_1+m_2)g$; the second separately gives $k_2(\hat Y_2-\hat Y_1)=-m_2g$. Hence
$$
\boxed{\hat X_1=\hat X_2=0,\qquad\hat Y_1=-\frac{(m_1+m_2)g}{k_1},\qquad\hat Y_2=-\frac{(m_1+m_2)g}{k_1}-\frac{m_2g}{k_2}.}
$$
Retain the imposed <constraint> $X_2=0$ and set $X_1=x_1$, $Y_1=\hat Y_1+y_1$, $Y_2=\hat Y_2+y_2$. Subtracting the equilibrium equations cancels all constant gravitational terms. For $q=(x_1,y_1,y_2)^T$, the <small-oscillation mass and stiffness matrices> are
$$
\boxed{M\ddot q+Kq=0,\quad M=\begin{pmatrix}m_1&0&0\\0&m_1&0\\0&0&m_2\end{pmatrix},\quad K=\begin{pmatrix}k_1+k_2&0&0\\0&k_1+k_2&-k_2\\0&-k_2&k_2\end{pmatrix}.}
$$
The horizontal reaction enforcing the constraint replaces the removed $X_2$ equation; it is not necessary to demand that the original unconstrained horizontal force on mass 2 vanish during motion. Because the <zero-natural-length spring> potentials are exactly quadratic, these displacement equations are actually exact within the ideal spring model, not merely first-order approximations.