Debiased Lasso

The debiased Lasso adds an approximate inverse-Gram correction to a Lasso estimate so that each corrected coordinate has an asymptotically linear expansion.

Let $\hat{\Sigma }=X^{T}X/n$ and use Nodewise Lasso with ${\lambda }_j>0$ and nonzero design columns. Form $c_j$ with entry $j$ equal to one and other entries $-{\hat{\gamma }}^{(j)}$, and put row $j$ of $\hat{\Theta }$ equal to $c_j^T/{\hat{\tau }}_j^2$. The nodewise-Lasso residual identity and the Karush-Kuhn-Tucker conditions show that the diagonal of $\hat{\Theta }\hat{\Sigma }$ equals one and its row-$j$ off-diagonal entries have magnitude at most ${\lambda }_j/{\hat{\tau }}_j^2$. For the Debiased Lasso,The displayed bound follows row by row from Holder inequality. If the noise is conditionally multivariate normal with covariance ${\sigma }^{2}I$, and the nodewise fits and tuning parameters depend only on $X$, the first term is conditionally normal with covariance ${\sigma }^2\hat{\Theta }\hat{\Sigma }{\hat{\Theta }}^T$.

A matrix $\Theta$ is an $\eta$-approximate inverse of a Gram matrix $\hat{\Sigma }$ in entrywise maximum norm whenThis turns the bias remainder of a Debiased Lasso into a product of $\eta$ and the coefficient ${\ell }^1$ error.

The nodewise Lasso regresses each design column on the remaining columns and uses its residual to estimate a column of the inverse covariance or inverse Gram matrix.

For Nodewise Lasso with objective $\Vert X_j-X_{-j}\gamma {\Vert }_2^2/(2n)+{\lambda }_j\Vert \gamma {\Vert }_1$, put $r_j=X_j-X_{-j}{\hat{\gamma }}^{(j)}$. Its Karush-Kuhn-Tucker conditions give $X_{-j}^T{r}_j/n={\lambda }_j{z}_j$ with $z_j\in \partial \Vert {\hat{\gamma }}^{(j)}{\Vert }_1$. HenceFor ${\lambda }_j>0$ this is positive whenever $X_j\ne 0$; if it were zero both nonnegative terms would vanish, forcing ${\hat{\gamma }}^{(j)}=0$ and $X_j=r_j=0$. Nonzero columns are therefore an essential hypothesis for division by this quantity.

Under sparsity, compatibility, and inverse-Gram estimation conditions, the standardized coordinatewise error of the Debiased Lasso converges in distribution to a standard normal distribution.

In the normal linear model, assume $\epsilon \sim N_n(0,{\sigma }^{2}I)$ is independent of $X$. Independent rows of $X$ with multivariate normal distribution $N_p(0,I_p)$ give a simple random design for Debiased Lasso. Let $p\to \infty$, $\log p=o(n)$ and $s\log p=o(\sqrt{n})$, with fixed positive noise scale. Choose the Lasso penalty and all Nodewise Lasso penalties as sufficiently large constant multiples of $\sqrt{\log p/n}$, including the noise scale in the former. Uniform sample cross-correlations are bounded by this scale, so zero is the nodewise solution with probability tending to one. Column squared norms divided by $n$ converge uniformly to one. To check the Compatibility condition for the Lasso directly, put $\hat{\Sigma }=X^{T}X/n$ and $\eta ={\max }_{j,k}|{\hat{\Sigma }}_{jk}-1_{\{j=k\}}|$. On the Lasso cone condition, $\Vert \delta {\Vert }_1\le 4\Vert {\delta }_S{\Vert }_1$, and the Cauchy-Schwarz inequality givesFor $s\ge 1$, entrywise concentration of a Gaussian Gram matrix gives $\eta \le D_0\sqrt{\log p/n}$ with probability tending to one for sufficiently large fixed $D_0$. The stated sparsity condition implies $s\eta \to 0$ on this event, so one can take ${\varphi }^2\ge 1/2$ eventually. The conditional Gaussian tail bound for the noise score and the Basic inequality for the Lasso then give $\Vert \hat{\beta }-{\beta }^0{\Vert }_1\le Cs\sqrt{\log p/n}$ with probability tending to one. Consequently the debiased-Lasso remainder bound is at most $As\log p/\sqrt{n}$ with probability tending to one, for a fixed constant $A$. These assumptions allow both $p$ and $s$ to grow, for example $p=n^2$, $s=\lfloor n^{1/4}\rfloor$.