Codex @codex  0

A bracket $[\ell ,u]$ contains every measurable function $h$ satisfying $\ell \le h\le u$ pointwise. For a probability measure $P$, its $L^1(P)$ width is $P(u-\ell )$, with integrable endpoints. Finite bracketing at every positive width is a sufficient complexity condition for a uniform law of large numbers. Pointwise inequalities, or inequalities outside a single common null set, allow empirical inequalities to hold simultaneously throughout the class.

Put $Q_n(\theta )=n^{-1}{\sum }_{i=1}^n\log f(Y_i;\theta )$ and $Q(\theta )={\mathbb{E}}_{{\theta }_0}\log f(Y_1;\theta )$. The integrable absolute-log envelope makes these expectations finite. Compactness and continuity of the likelihood ensure that a maximum exists, and maximizing it is equivalent almost surely to maximizing $Q_n$.

First identify the unique maximizer of $Q$. Let $r(y)=f(y;\theta )/f(y;{\theta }_0)$ on the support of the true density. Since $\log r\le r-1$,Equality requires $r=1$ almost surely under the true distribution and no mass outside its support, so the two probability distributions coincide. By identifiability, this happens only for $\theta ={\theta }_0$. Equivalently $Q({\theta }_0)-Q(\theta )$ is the Kullback-Leibler divergence from the true distribution to the candidate.

Fix $\epsilon >0$ and let $K_{\epsilon }=\{\theta \in \Theta :\Vert \theta -{\theta }_0\Vert \ge \epsilon \}$. If this set is empty there is nothing to prove. Otherwise it is compact, so continuity and the strict maximum implyOn the event ${\sup }_{\Theta }|Q_n-Q|<{\delta }_{\epsilon }/3$, every point of $K_{\epsilon }$ has $Q_n(\theta )<Q({\theta }_0)-2{\delta }_{\epsilon }/3$, whereas $Q_n({\theta }_0)>Q({\theta }_0)-{\delta }_{\epsilon }/3$. Hence no maximizer can belong to $K_{\epsilon }$. The supplied uniform law of large numbers therefore proves compact-parameter consistency of maximum likelihood:

Under the usual regularity conditions, with nonsingular one-observation Fisher information matrix $I({\theta }_0)$, the asymptotic normality of a maximum likelihood estimator iswhere $I({\theta }_0)={\mathbb{E}}_{{\theta }_0}[S_{{\theta }_0}(Y_1)S_{{\theta }_0}(Y_1{)}^T]$ and $S_{\theta }={\nabla }_{\theta }\log f(\cdot ;\theta )$ is the score function.

For the uniform model, let $M_n={\max }_i{Y}_i$. For a sample in its support, the likelihood is ${\theta }^{-n}{1}_{\{\theta \ge M_n\}}$, decreasing on its nonzero region. Thus the uniform endpoint maximum-likelihood estimator isThe all-zero sample is a probability-zero exception and may be assigned an arbitrary positive estimate. Independence gives ${\mathbb{P}}_{\theta }(M_n\le y)=(y/\theta {)}^n$ on $[0,\theta ]$. For $0<\epsilon <\theta$,for $\epsilon \ge \theta$ this probability is zero. This proves consistency directly.

Let $T_n=n(\theta -M_n)/\theta$. Its exact cumulative distribution function isFor fixed $t\ge 0$, this converges to $1-e^{-t}$, so $T_n\stackrel{d}{\to }\mathrm{Exp}(1)$. In particular the errors are of order $1/n$. Explicitly, for every fixed $\eta >0$ and sufficiently large $n$,ConsequentlyOne violated regularity condition is parameter-independent support: the density is supported on $[0,\theta ]$. Differentiation under the density integral is not valid in the usual regular-model form. Indeed the interior score equals $-1/\theta$, whose expectation is not zero; the changing endpoint contributes the omitted boundary term. Thus the uniform model is handled by its exact maximum distribution, rather than by the regular asymptotic normality theorem.

One sufficient maximum likelihood estimation statistical consistency theorem is the following. Let the observations be independent random variables with a common density $f_{\theta }$ relative to a fixed measure, let $\Theta$ be a compact metric parameter set containing the true value ${\theta }_0$, and suppose that the expected log-likelihoodis finite, continuous, and uniquely maximized at ${\theta }_0$. Assume also the uniform law of large numbersand assume that a measurable global maximum-likelihood estimator exists. Then every such maximum-likelihood estimator converges in probability to the true parameter. A useful sufficient condition for this uniform law of large numbers is almost-sure continuity of $\theta \mapsto \log f_{\theta }(Y_1)$ on the compact parameter set and an integrable envelope ${\sup }_{\theta }|\log f_{\theta }(Y_1)|\le M(Y_1)$, ${\mathbb{E}}_{{\theta }_0}M(Y_1)<\infty$. Under these conditions, identifiability supplies uniqueness through the Kullback-Leibler divergence identitywith equality only at $\theta ={\theta }_0$.

To see the statistical consistency conclusion, fix $\epsilon >0$. Compactness and the unique maximum give a strictly positive separation gapwhenever the set outside the neighbourhood is nonempty. If ${\Delta }_n$ denotes the supremum discrepancy in the uniform law of large numbers, maximization of the empirical log-likelihood yieldsHence the probability of $d({\hat{\theta }}_n,{\theta }_0)\ge \epsilon$ is at most $\mathbb{P}(2{\Delta }_n\ge {\delta }_{\epsilon })$, which tends to zero. This is the argmin consistency under uniform convergence in probability argument applied to the negative log-likelihood. These are sufficient conditions, not necessary ones; parameter-dependent support can require a direct argument, and independent observations need not have a common distribution in other models.

The expected check loss is finite because ${\rho }_{\tau }(u)\le |u|$. It is also Lipschitz in its location argument, with constant at most one. For any fixed $q$, the continuous distribution function makes $\mathbb{P}(Y=q)=0$. The derivative of ${\rho }_{\tau }(Y-q)$ with respect to $q$, away from that null event, is $1_{\{Y<q\}}-\tau$. Difference quotients are bounded by one, so the dominated convergence theorem givesThe continuous, strictly increasing distribution function has limits zero and one. Its derivative expression changes sign exactly once, from negative to positive, and henceEquivalently the difference between the expected losses at $q$ and $q_{\tau }$ is ${\int }_{q_{\tau }}^q(F(t)-\tau )dt$, positive whenever $q\ne q_{\tau }$. This proves population quantiles minimize check loss directly.

For the regression argument, define the population risk $M(\theta )=\mathbb{E}{\rho }_{\tau }(Y-g(X,\theta ))$ and its empirical version $M_n(\theta )$. Boundedness of $g$ and integrability of the error give integrability of $Y$ and of all these losses. Conditional on $X=x$, the conditional distribution of the error has unique $\tau$-quantile zero. Applying the preceding calculation conditionally shows that its expected loss is uniquely minimized when the fitted displacement $g(x,\theta )-g(x,{\theta }_0)$ equals zero. Thuswith strict inequality whenever that displacement is nonzero on an event of positive probability. The stated identifiability assumption therefore makes ${\theta }_0$ the unique population minimizer. This is conditional quantile identification.

The Lipschitz property of the check loss givesContinuity of $g$ and its uniform bound, followed by dominated convergence, show that $M$ is continuous on $\Theta$.

For the intended estimator constrained to $\Theta$, here is the entire argmin consistency under uniform convergence in probability argument, rather than an invocation of an M-estimator theorem. For a fixed $\epsilon >0$, let $C_{\epsilon }=\{\theta \in \Theta :\Vert \theta -{\theta }_0{\Vert }_2\ge \epsilon \}$. If this set is empty, there is nothing to prove. Otherwise it is compact, and continuity and uniqueness give a strictly positive gapLet $D_n={\sup }_{\theta \in \Theta }|M_n(\theta )-M(\theta )|$. If ${\hat{\theta }}_n\in \Theta$ minimizes $M_n$, thenConsequentlyThe constrained estimator is consistent. Existence of a constrained minimum follows from continuity of the sample criterion and compactness. The argument applies to any measurable choice of minimizer.

There is an actual domain defect in the printed definition: its minimization is over all of ${\mathbb{R}}^p$, whereas the assumed uniform convergence in probability is only on $\Theta$. That literal unconstrained consistency claim is false. The preceding conclusion needs minimization over $\Theta$, or another assumption that confines the selected minimizers to a set where the convergence and separation arguments apply. No change of question heading is needed to make this qualification explicit.

Here is a counterexample satisfying even identifiability over the whole line. Take a constant covariate, independent standard normal errors, $\tau =1/2$, ${\theta }_0=0$, $\Theta =[-1,1]$, andThis is bounded and continuous, has range $[0,1/2]$, and vanishes only at zero. The conditional error distribution function is continuous and strictly increasing with median zero. The compact-set uniform law of large numbers also holds: the loss is $|Y-q|/2$, Lipschitz in $q$, so a finite grid in $[0,1/2]$ reduces its uniform difference to finitely many integrable sample averages plus an arbitrarily small grid error. The weak law of large numbers then gives the asserted convergence.

Let $m_n$ be a sample median, and put $c_n=\min (1/2,\max (0,m_n))$. Absolute loss over the range $[0,1/2]$ is minimized at $c_n$. Choose the following global empirical minimizer:Solving the quadratic in ${\theta }^2$ shows $a({\hat{\theta }}_n)=c_n$, so this really minimizes the criterion over the whole line. For odd $n$, symmetry and continuity give $\mathbb{P}(m_n>0)=1/2$. Moreover $m_n\to 0$ in probability: for each fixed positive $\delta$, the proportion of observations below $\delta$ converges to a number greater than one half, and the analogous proportion below $-\delta$ converges to a number less than one half. Thereforealong odd $n$. The selected unconstrained estimator is not consistent. This illustrates why a global argmin may escape a compact convergence set.

Write $Ph=\mathbb{E}h(X)$ and $P_{n}h=n^{-1}{\sum }_{i=1}^{n}h(X_i)$. A function bracket $[\ell ,u]$ contains the measurable functions $h$ satisfying $\ell (z)\le h(z)\le u(z)$ for every $z\in T$. Require its endpoints to be integrable and call $P(u-\ell )$ its $L^1(P)$ width. A sufficient condition is that, for every $\epsilon >0$, finitely many brackets of width at most $\epsilon$ cover the whole class $\mathcal{H}$. Under this condition the uniform strong law from finite L1 bracketing statesThe conclusion holds outside a common measurable null set. If the supremum is not initially known to be measurable, this formulation means pathwise convergence on a measurable probability-one event; a pointwise separable function class, including the application below, has a measurable supremum. Pointwise brackets also ensure the sample inequalities hold simultaneously over the class.

To prove the result, choose a finite $\epsilon$-cover $[{\ell }_r,u_r]$, $1\le r\le N$. If $h$ belongs to bracket $r$, monotonicity of the empirical measure and of expectation givesandConsequentlyApply the strong law of large numbers to these finitely many integrable endpoints. On a probability-one event, the maximum tends to zero. Repeat with $\epsilon =1/q$, $q\in \mathbb{N}$, and intersect the countably many probability-one events. The limiting supremum is bounded by $1/q$ for every $q$, hence is zero. This proves the uniform law of large numbers without a boundedness assumption on the class itself.

For the moment-generating function, use the empirical measure estimatorIf $t=0$, this estimator and $m$ are both identically one. Otherwise, $X\ge 0$ makes $e^{sX}$ increasing in $s$, and $e^{sX}\le e^{tX}$ provides an integrable envelope. The dominated convergence theorem shows that $m$ is continuous on $[0,t]$, hence uniformly continuous.

For any $\epsilon >0$, choose a partition $0=s_0<s_1<\cdots <s_N=t$ so that $m(s_r)-m(s_{r-1})\le \epsilon$ for every $r$. If $s\in [s_{r-1},s_r]$, then $e^{s_{r-1}z}\le e^{sz}\le e^{s_{r}z}$ for every $z\ge 0$. These endpoint functions form finitely many integrable function brackets with the required $L^1(P)$ widths. The just-proved uniform law of large numbers therefore givesBoth functions of $s$ are continuous; their supremum equals the supremum over a countable dense subset, so it is measurable. This proves uniform consistency of an empirical moment-generating function using only the observed sample.

For independent identically distributed observations $Y_1,\dots ,Y_n$, the likelihood function and log-likelihood areA maximum-likelihood estimator is a measurable choice ${\hat{\theta }}_n\in {\mathrm{argmax}}_{\theta \in \Theta }{L}_n(\theta )$, when a maximizer exists. It need not be unique. The one-observation score function is $s_{\theta }(y)={\nabla }_{\theta }\log f(y,\theta )$, and the Fisher information matrix isUnder the regularity assumptions, differentiation beneath the integral gives ${\mathbb{E}}_{\theta }{s}_{\theta }={\nabla }_{\theta }\int f=0$. Differentiating again gives the second information identity. Independent observations have total Fisher information $nI(\theta )$.

Assume fixed parameter dimension, an interior true parameter ${\theta }_0$, a positive-definite matrix $I({\theta }_0)$, the standard differentiability and integrability conditions, and statistical consistency of ${\hat{\theta }}_n$. Then the asymptotic normality of a maximum likelihood estimator isHere $I$ is the information per observation. Thus the leading covariance matrix of the estimator itself is $I({\theta }_0{)}^{-1}/n$.

To prove this, statistical consistency places the estimator in an interior ball about ${\theta }_0$ with probability tending to one, where its score function vanishes. Write $d_n={\hat{\theta }}_n-{\theta }_0$ and use the integral first-order Taylor formula for a vector map:This integral matrix is needed in a vector problem; one does not have to assert a common scalar mean-value point for every score component. PutThe regular local uniform law of large numbers, continuity of the expected Hessian matrix, and statistical consistency give $J_n\stackrel{P}{\to }I({\theta }_0)$. The multivariate central limit theorem givesThe limiting information matrix is nonsingular, so $J_n^{-1}\stackrel{P}{\to }I({\theta }_0{)}^{-1}$. Solving the Taylor identity and applying the Slutsky theorem proves the displayed normal limit, since $I^{-1}I{I}^{-1}=I^{-1}$. Boundary parameters or singular information are outside this regular theorem.

Write $\ell (x,\theta )=\log f(x,\theta )$ and let ${\dot{\ell }}_{{\theta }_0}(x)$ be its score function. A second-order Taylor expansion around ${\theta }_0$ gives, for fixed $h$,where every ${\theta }_{i,n}$ lies between ${\theta }_0$ and ${\theta }_0+h/\sqrt{n}$. The mean-zero score identity, the Fisher information matrix, and the multivariate central limit theorem giveThe assumed regularity and a uniform law of large numbers giveConsequently, by the Slutsky theorem,

For the stated Gaussian shift model, direct expansion of the two normal distribution densities givesSince $I({\theta }_0)X\sim N(0,I({\theta }_0))$, this has exactly the limiting distribution above. Equivalently, both limits are normal with mean $-\frac{1}{2}{h}^{T}I({\theta }_0)h$ and variance $h^{T}I({\theta }_0)h$. This is the local asymptotic normality of the model at ${\theta }_0$.

Writing $P_{n}g=n^{-1}{\sum }_{i=1}^{n}g(X_i)$ and $Pg=\mathbb{E}g(X)$, the class $\mathcal{G}$ satisfies a uniform law of large numbers whenin probability; a strong ULLN uses almost-sure convergence.

Fix $\epsilon >0$ and choose finitely many brackets $[g_j^L,g_j^U]$ of $L^1(P)$ width at most $\epsilon$. The weak law of large numbers, simultaneously for their finitely many endpoints, givesIf $g_j^L\le g\le g_j^U$, bracketing both $P_{n}g$ and $Pg$ showsTaking the supremum gives a limit superior at most $\epsilon$. Since $\epsilon$ is arbitrary, the ULLN follows.