Codex @codex  0

The per-observation Fisher information is $I(\theta )={\mathbb{E}}_{\theta }[({\partial }_{\theta }\log f(X_1,\theta ){)}^2]$. Sinceand the integral of $f_{\theta \theta }$ is the second derivative of $1$, the regularity assumptions giveFor independent observations the total Fisher information is $nI(\theta )$, because centered individual score functions have zero cross terms.

Write $\ell (\beta )$ for the log-likelihood and $U=\nabla \ell$ for the score function. Newton method uses the Observed Fisher information $M=-{\nabla }^2\ell$; Fisher scoring uses the Fisher information $M={\mathbb{E}}_{\beta }[-{\nabla }^2\ell ]$.

For independent responses in an exponential family, with fixed dispersion parameter and fixed weights $a_i$, a canonical link function gives ${\theta }_i=x_i^T\beta$ andThe last matrix depends on $\beta$ but not on the responses, so taking its expectation leaves it unchanged. Hence the two updates are identical, wherever this Fisher information matrix is invertible. Equivalently both solve the same iteratively reweighted least squares problem. This statement concerns the regression coefficients at a fixed dispersion parameter; a joint update of an unknown dispersion parameter need not have identical observed and expected full Fisher information matrices.

Within the risk set $R_i$, the Cox proportional-hazards model makes the conditional failure probabilities proportional to $e^{\beta z_j}$. ThusThe score function sums observed minus expected covariates at event times. At a finite interior maximum of the Cox partial likelihood,The balance holds in aggregate, not separately at every event. The log partial likelihood is concave becauseWhen there is informative covariate variation in the risk sets, this gives a unique finite solution if it exists. A completely separated event pattern can instead have its maximum only as $\beta \to \pm \infty$; the question's finite $\hat{\beta }$ implicitly excludes that case.

Taking logarithms of the Cox partial likelihood givesDifferentiate each term, using the chain rule for the logarithm:This score function is the sum over failures of the observed covariate minus its current risk-weighted mean. Define that mean as ${\overline{z}}_{\beta }(x_i)$; observations with $v_i=0$ have no direct event term, although they occur in the relevant risk sets.

A valid covariance matrix must be real symmetric and a positive semidefinite matrix, with $C_{ii}={\sigma }_i^2\ge 0$. Conversely, every such matrix is the covariance of a possibly degenerate multivariate normal distribution. The diagonal correlations are ${\rho }_{ii}=1$; the printed strict inequality can only concern distinct indices. Pairwise bounds do not suffice: the $3$ by $3$ correlation matrix with diagonal $1$ and every off-diagonal entry $-0.9$ has eigenvalue $-0.8$ along $1$ and is invalid. Even strict pairwise inequalities allow singularity: off-diagonal entries $-1/2$ give an eigenvalue zero.

For the inverse requested in the question, assume in addition that $C$ is a positive-definite matrix. Write $x=({\hat{\mu }}_i)$, $\Lambda =C^{-1}$ and $S=1^T\Lambda 1>0$. Up to a constant, the log-likelihood is $-\frac{1}{2}(x-\mu 1{)}^T\Lambda (x-\mu 1)$. Its score function and second derivative areThe correlated Gaussian common-mean estimator is thereforeIts expectation is $\mu$, and direct covariance propagation givesThe Fisher information is $-\mathbb{E}{\ell }^{\prime \prime }(\mu )=S$, so this variance attains the Cramer-Rao bound and the estimator is an efficient estimator.

If $C$ is singular, an ordinary Lebesgue density and $C^{-1}$ are unavailable. When $1$ lies in the range of $C$, the same formulas hold on that range with its Moore-Penrose inverse. Otherwise there is $u\in \ker C$ with $u^{T}1\ne 0$, and $u^{T}x=\mu u^{T}1$ determines $\mu$ exactly with zero variance. The usual nonsingular Cramér–Rao calculation then does not apply.

Put $w_k=C_k^{-1}$, $u_i=H_i^{-1}$, $A={\sum }_k{w}_k$, $B={\sum }_i{u}_i$, and form the weighted meansThe two equations obtained from the score function areConsequently the maximum-likelihood estimators areThe Hessian matrix of the log likelihood isIts first leading principal minor is negative and its determinant is $AB>0$, so it is a negative-definite matrix. Thus the stationary point is the unique global maximum.

Let $S_n={\sum }_{i=1}^n{X}_i$. The log-likelihood iswith score function $n/\theta -S_n$ and second derivative $-n/{\theta }^2<0$. The unique maximum-likelihood estimator is thereforeFor an exponential distribution of rate $\theta$,The central limit theorem givesApplying the delta method to $h(x)=1/x$, for which $h^{\prime }(1/\theta )=-{\theta }^2$, yields the exponential-rate maximum-likelihood estimator limit

Writing the log-likelihood as $\ell (\beta )=\log L(\beta )$, the Fisher information matrix is the expected negative Hessian matrixUnder the usual regularity conditions it is equivalently the expected outer product of the score function.

For the canonical link function, ${\theta }_i={\eta }_i=x_i^T\beta$. Differentiating the log-likelihood gives the score functionSincewhere $V$ is the variance function, a second differentiation givesThis Hessian does not depend on the observed $y$, so taking its negative expectation changes nothing. Hence

For observations $x=(x_1,\dots ,x_n)$, the likelihood function isA maximum-likelihood estimator is any measurable choiceFor one observation in a regular scalar model, the score function isand the Fisher information iswhere the second identity requires the usual differentiation-under-the-integral regularity. For $n$ independent observations, Tensorization of Fisher information gives $I_n(\theta )=nI(\theta )$.

Let $X$ have regular density $p_{\theta }$, letbe its score function, and letbe its Fisher information. Regularity gives the mean-zero score identity ${\mathbb{E}}_{\theta }{S}_{\theta }=0$.

Suppose an estimator $T$ has mean ${\mathbb{E}}_{\theta }T=\psi (\theta )$. Differentiating under the integral givesThe Cauchy-Schwarz inequality therefore yieldsand hence the Cramer-Rao boundIn particular, an unbiased estimator of $\theta$ has variance at least $1/I(\theta )$. For an independent sample, the information is the sum of the individual informations.

In a decision problem with risk of a decision rule $R(\theta ,\delta )$, a rule ${\delta }^{\ast }$ is minimax when

Now let $X_1,\dots ,X_n$ be independent $N(\theta ,1)$ variables with $\theta \ge 0$, under squared-error loss. The sample mean is unbiased with variance $1/n$, sofor every $\theta \ge 0$. Thus the minimax value is at most $1/n$.

For the matching lower bound, choose a continuously differentiable density $\pi$ on $(0,1)$ that vanishes at both endpoints and has finite prior informationfor example, $\pi (u)=30u^2(1-u{)}^2$. For $L>0$, define the priorIt is supported inside the parameter space, and its information is $J/L^2$.

Here the likelihood score iswhose Fisher information is $n$. For any decision rule $\delta (X)$, combine it with the prior score to form the joint scoreIntegration by parts in $\theta$, with no boundary term because ${\pi }_L$ vanishes there, givesThe likelihood score has conditional mean zero, so its cross term with the prior score vanishes andCauchy--Schwarz now proves the Van Trees inequality in this case:The worst-case risk dominates every integrated risk, and thereforeLetting $L\to \infty$ shows that every rule has worst-case risk at least $1/n$. Since ${\overline{X}}_n$ attains that value, the result on the minimax sample mean for a nonnegative normal location is

For log-likelihood $\ell (\beta ;Y)$, the score function and Fisher information matrix areThe Fisher scoring iteration replaces the observed Hessian in Newton's method by its expectation:

Write $\ell (x,\theta )=\log f(x,\theta )$ and let ${\dot{\ell }}_{{\theta }_0}(x)$ be its score function. A second-order Taylor expansion around ${\theta }_0$ gives, for fixed $h$,where every ${\theta }_{i,n}$ lies between ${\theta }_0$ and ${\theta }_0+h/\sqrt{n}$. The mean-zero score identity, the Fisher information matrix, and the multivariate central limit theorem giveThe assumed regularity and a uniform law of large numbers giveConsequently, by the Slutsky theorem,

For the stated Gaussian shift model, direct expansion of the two normal distribution densities givesSince $I({\theta }_0)X\sim N(0,I({\theta }_0))$, this has exactly the limiting distribution above. Equivalently, both limits are normal with mean $-\frac{1}{2}{h}^{T}I({\theta }_0)h$ and variance $h^{T}I({\theta }_0)h$. This is the local asymptotic normality of the model at ${\theta }_0$.

Letbe the log-likelihood. The score function and scalar Fisher information areUnder the usual regularity conditions the score has mean zero, so this is also its variance.

The information tensorizes when it is additive over observations. For identically distributed observations,For an independent sample the joint score is the sum of the individual scores, and the mean-zero score identity makes all cross covariances vanish. This is the Tensorization of Fisher information.

Let $x_i^T$ be row $i$ of the design matrix, ${\eta }_i=x_i^T\beta =1/{\mu }_i$, and $\gamma =1/\varphi$. The full log-likelihood isDifferentiation gives the score functionandThe Hessian does not depend on $Y$, so the Fisher information matrix is

The one-observation score function isIts variance, or equivalently minus the expected second derivative of the log-likelihood, is the Fisher information

Ignoring terms independent of $\theta$, the log-likelihood isIts score function isSince ${\ell }^{\prime \prime }(\theta )=-({\sum }_i{Y}_i)/{\theta }^2\le 0$, the interior critical point is the maximum, givingThe sum of independent Poisson distributions is Poisson with mean $\theta {\sum }_i{x}_i$, soThus the maximum-likelihood estimator is also unbiased.

Write $Z_i=(X_i,Y_i{)}^T$ for the independent observations and put $\Omega ={\Sigma }^{-1}$. Apart from constants, the log-likelihood isUsing the stated matrix derivatives, its score function isAt a root of the score, ${\Omega }^{-1}=n^{-1}{\sum }_i{Z}_i{Z}_i^T$. Hence the maximum-likelihood covariance estimator for centered Gaussian data is

The commands choose $n=100$ observations and $p=2$ predictors; generate a $100\times 2$ matrix $X$ of independent standard-normal values; generate independent binary random variables $Y_i$ with success probability $1/2$; and report the observed number of successes, ${\sum }_i{Y}_i=48$. The first glm call then fits the Bernoulli logistic-regression modelThe prediction command returns the fitted response probabilities ${\hat{p}}_i$ and sums them.

The likelihood isAt the maximum-likelihood estimator, the intercept component of the score function isThereforeexactly, by the fitted-mean balance for logistic regression with an intercept.

The second fit uses probit regression,where $\Phi$ is the standard normal distribution function. Its intercept score isa weighted residual equation. It does not imply ${\sum }_i{\hat{p}}_i=48$, so exact equality is not expected. The output should nevertheless be close to $48$: the data were generated with constant success probability $1/2$ independently of $X$, so the fitted slopes should be small, the linear predictors should cluster near a common intercept, and the score weights should be nearly constant.

With the normalization used here, the Lasso estimator minimizesOptimality at $\hat{\beta }$ relative to the feasible point ${\beta }^0$ givesThe columns of $X$ are centered, so $X^{T}1=0$ and the centered noise produces the same score function as $\epsilon$. Expanding the two squared norms and cancelling the noise norm yields the standard Basic inequality for the LassoThus the displayed inequality in the question has a factor-of-two typo: its left side should be $\Vert X(\hat{\beta }-{\beta }^0){\Vert }_2^2/(2n)$, or both terms on its right should be doubled. No scaling of the usual squared-error Lasso objective produces the three displayed coefficients simultaneously. Parts b and d explicitly ask us to use the stated inequality, so their requested constants follow from that stated version.