Codex @codex  0

By the spectral theorem, write $V=U\mathrm{diag}(v_1,\dots ,v_p)U^T$, where $U$ is orthogonal and each $v_j>0$. Choose the whitening transformationA linear image of a multivariate normal vector is multivariate normal, and $LV{L}^T=I$. Thus $Z_i=L(Y_i-\mu )$ are independent $N_p(0,I)$ random vectors. Their whole joint distribution is independent of $\mu ,V$.

Define $\bar{Z}=n^{-1}{\sum }_i{Z}_i$ and $S_Z=n^{-1}{\sum }_i(Z_i-\bar{Z})(Z_i-\bar{Z}{)}^T$. These satisfyThereforeThe right side is a function of independent standard multivariate normal observations only, so its distribution depends on $n,p$ but not on $\mu ,V$. This is the same affine invariance of Hotelling's statistic that permits multivariate mean tests with unknown covariance. The proof concerns the jointly transformed mean and covariance; whitening the mean alone would not establish the conclusion. Inverses exist almost surely under $n>p$. If $V$ is singular, no square matrix can whiten it to $I$, so nonsingularity of the population covariance is necessary as well.

Write the expected value as ${\mu }^T=({\mu }_1^T,{\mu }_2^T)$ and assume the relevant covariance matrix $V_{22}$ is invertible. Remove the linear predictor from the first block by definingA linear image of a multivariate normal vector is again multivariate normal. Direct covariance calculation givesThus $Y$ and $X_2$ are independent: uncorrelated jointly Gaussian variables are independent. Expanding the covariance matrix of $Y$, using $V_{21}=V_{12}^T$, givesConditioning on $X_2=x_2$ translates $Y$ by the fixed vector ${\mu }_1+V_{12}{V}_{22}^{-1}(x_2-{\mu }_2)$ and leaves its covariance matrix unchanged. Consequently the conditional multivariate normal distribution isThe conditional covariance is the Schur complement and does not depend on $x_2$.

For equal prior probabilities, $b=\frac{1}{2}{a}^T({\mu }_1+{\mu }_2)$. The projected score is normal under either class because it is a linear image of a multivariate normal vector, and its variance isMoreover $a^T{\mu }_1-a^T{\mu }_2={\delta }^2$, so the midpoint threshold lies ${\delta }^2/2$ from either projected mean. HenceStandardizing the two normal distributions, with $\Phi$ the standard normal distribution function, givesThus the equal-covariance Gaussian classification error is $p(\delta )=\Phi (-\delta /2)$. The Mahalanobis distance $\delta$ between the class means is their separation in units of within-class noise. It yields $p\to 1/2$ as $\delta \downarrow 0$ and $p\to 0$ as $\delta \to \infty$. The stipulated $\delta >0$ means the two class means are distinct.

Take the coordinate-selection matrixThen $A^{T}X=X_1$, $A^T\mu ={\mu }_1$ and $A^T\Sigma A={\Sigma }_{11}$. The permitted result on a linear image of a multivariate normal vector therefore givesThis is the marginal distribution obtained by discarding the other coordinates. The conclusion holds also for a positive-semidefinite marginal covariance: in that case the Gaussian law may be degenerate and need not have a full-dimensional density. If the original covariance is positive definite, its principal block ${\Sigma }_{11}$ is positive definite as well.

For every nonzero $v\in {\mathbb{R}}^{n-p}$, full column rank of $L$ gives $Lv\ne 0$, and henceThus the Gram matrix $L^{T}L$ is a positive-definite matrix and is invertible. Using $BA=0$ and $A=L{L}^T$,so

A linear image of a multivariate normal vector is again multivariate normal. Therefore the stacked vector has mean $(B\mu ,L^T\mu {)}^T$ and covarianceIn particular,The two diagonal blocks are positive-definite by the rank assumptions. The two blocks are jointly Gaussian and have zero cross covariance. The fact that uncorrelated jointly Gaussian variables are independent makes $BY$ and $L^{T}Y$ independent. Sinceis a measurable function of the second block, independence is preserved under this transformation. This proves Gaussian independence of linear and quadratic statistics:

The regression residual vector is $e=Y-X\hat{\beta }=(I-H)Y$, where $H=X(X^{T}X{)}^{-1}{X}^T$ is the hat matrix. Both $H$ and $I-H$ are symmetric idempotent matrices, and $(I-H)X=0$. Therefore $e=(I-H)\epsilon$ and the linear image of a multivariate normal vector givesThis multivariate normal distribution is singular, with rank $n-p$: fitted-residual orthogonality forces $X^{T}e=0$. Specifically, $\mathrm{Var}(e_i)={\sigma }^2(1-h_{ii})$ and $\mathrm{Cov}(e_i,e_j)=-{\sigma }^2{h}_{ij}$ for distinct observations. Thus regression residuals are generally neither independent nor identically distributed, even though the model errors are. Also $\mathrm{RSS}/{\sigma }^2\sim {\chi }_{n-p}^2$ and is independent of $\hat{\beta }$, which justifies the standard errors and exact Student t-tests below.

The Gaussian likelihood has log-likelihood, apart from no omitted parameter-dependent terms,For fixed ${\sigma }^2$, maximizing it is ordinary least squares. Differentiating with respect to $\beta$ gives the normal equation $X^{T}X\hat{\beta }=X^{T}Y$. Full column rank makes the quadratic strictly convex, soAt this value, differentiating with respect to $v={\sigma }^2$ gives $-n/(2v)+\mathrm{RSS}/(2v^2)=0$, soFor positive RSS this is the global likelihood maximum in $v>0$. RSS is positive almost surely under the model since $p<n$; an exactly zero observed RSS instead gives a boundary supremum as $v\downarrow 0$.

As a linear image of a multivariate normal vector, the coefficient estimate satisfiesIndeed, $\hat{\beta }-\beta =(X^{T}X{)}^{-1}{X}^T\epsilon$, and direct matrix multiplication gives its covariance matrix.

For a convex function $f$, its subdifferential at $x$ is the set of supporting slopesThe subgradient optimality condition isIndeed, the defining inequality with $g=0$ is exactly the global minimum inequality. For the subdifferential of the L1 norm, the coordinates vary independently:

Let $r=Y-X{\hat{\beta }}_{\lambda }$. The Karush-Kuhn-Tucker conditions for Lasso give a vector $z\in \partial \Vert {\hat{\beta }}_{\lambda }{\Vert }_1$ such that $X^{T}r/n=\lambda z$. If ${\hat{\sigma }}_{\lambda }=\Vert r{\Vert }_2/\sqrt{n}>0$, the unsquared residual norm is differentiable at this fit, with gradientUsing the subdifferential sum rule, choose $\gamma (\lambda )=\lambda /{\hat{\sigma }}_{\lambda }$ to obtainConvexity now proves the tuning relation between Lasso and square-root Lasso:The positive residual assumption is needed both for this gradient and for the quotient defining the tuning parameter.

For the final score calculation, use the usual fixed-design interpretation of the normal linear model: regard $X,Z$ and the tuning parameter $\gamma$ in the regression of $Z$ as fixed, and choose that regression's minimizer using only those inputs. Its nonzero residual $R=R_{\gamma }$ then has the Square-root Lasso optimality conditionExpanding the response residual givesSince $R/\Vert R{\Vert }_2$ is a fixed unit vector and $\epsilon \sim N_n(0,{\sigma }^2{I}_n)$, a linear image of a multivariate normal vector gives $W\sim N(0,{\sigma }^2)$. Finally, Holder inequality and the residual score bound yieldThis is the residual-score decomposition from square-root Lasso. The normality statement also holds conditionally on a design independent of the noise. If $\gamma$ were chosen from the response noise, for example through the preceding $\gamma (\lambda )$ formula, the direction $R$ would require a separate independence argument; the final calculation treats $\gamma$ as fixed.

Condition on the information available before time $n$. The current wealth is then fixed, while ${\xi }_n\sim N(b,\Sigma )$ is independent of that information. PutFor any portfolio $\theta$, use the inner product induced by the positive-definite matrix $\Sigma$ to writeThe linear image of a multivariate normal vector shows that the two portfolio returns are jointly normal, andThus independence of uncorrelated jointly normal variables makes the residual return ${\eta }^T{\xi }_n$ independent of the aligned return; it also has expected value zero.

The continuation value $V(n,\cdot )$ is concave by part (c). Conditional Jensen inequality therefore shows that adding the independent centered residual cannot improve the objective:where $m=(1+r)X_{n-1}$. Since the optimal portfolio is unique, its residual must be zero.

It remains to determine the sign. If $\lambda <0$, the portfolios $\lambda {\Sigma }^{-1}b$ and $-\lambda {\Sigma }^{-1}b$ have the same return variance, namely ${\lambda }^{2}B$, while their means are $\lambda B$ and $-\lambda B$. The latter return has the distribution of the former plus the positive constant $-2\lambda B$. Since $V(n,\cdot )$ is increasing, replacing $\lambda$ by $-\lambda$ cannot reduce the objective, contradicting uniqueness. Hence $\lambda \ge 0$.

Applying this conditional argument at every time gives nonnegative, past-measurable random variables ${\lambda }_n$ such thatThis is the Gaussian one-fund theorem.