Solution (source code)

= Solution

Let $\Xi_{ik}=\langle X_i,B_k\rangle$ and define the <roughness penalty matrix>
$$
\Omega_{jk}=\langle B_j'',B_k''\rangle
=\int_0^1B_j''(t)B_k''(t)\,dt.
$$
For $c=(c_1,\ldots,c_K)^{\mathsf T}$, the loss is the quadratic function
$$
L(c)=\lVert Y-\Xi c\rVert_2^2+\rho c^{\mathsf T}\Omega c.
$$
Its <normal equations> are
$$
(\Xi^{\mathsf T}\Xi+\rho\Omega)c=\Xi^{\mathsf T}Y.
$$
Whenever $\Xi^{\mathsf T}\Xi+\rho\Omega$ is positive definite, the unique minimizer is
$$
\widehat c
=(\Xi^{\mathsf T}\Xi+\rho\Omega)^{-1}\Xi^{\mathsf T}Y,
\qquad
\widehat\beta(t)=\sum_{k=1}^K\widehat c_kB_k(t).
$$
For the usual choice $\rho\geq0$, the penalty matrix is positive semidefinite, so full column rank of $\Xi$ suffices. Since the question permits arbitrary real $\rho$, a sufficiently negative value can make the quadratic form indefinite; then the loss is unbounded below and no minimizer exists. In the singular positive-semidefinite case, the <Moore-Penrose inverse> describes the minimum-norm solution whenever the normal equations are consistent.