Uniform law of large numbers (ULLN)

A class $\mathcal{G}$ satisfies a uniform law of large numbers when its empirical means converge uniformly to their expectations:

A measurable-function class is pointwise separable if it has a countable subclass such that every member is the pointwise limit of a sequence from that subclass. With an integrable common envelope, dominated convergence theorem makes both empirical averages and population expectations converge along each such sequence. Thus the supremum of the absolute empirical errors equals the supremum over the countable subclass and is measurable. A finite integrable bracketing cover provides an integrable envelope by taking the maximum absolute value of all its endpoints.

A bracket $[\ell ,u]$ contains every measurable function $h$ satisfying $\ell \le h\le u$ pointwise. For a probability measure $P$, its $L^1(P)$ width is $P(u-\ell )$, with integrable endpoints. Finite bracketing at every positive width is a sufficient complexity condition for a uniform law of large numbers. Pointwise inequalities, or inequalities outside a single common null set, allow empirical inequalities to hold simultaneously throughout the class.

A measurable function $M$ is an integrable envelope of a function class for a class $\mathcal{H}$ under a probability law $P$ if $|h(x)|\le M(x)$ for every $h\in \mathcal{H}$ and $PM<\infty$. Such an envelope permits the dominated convergence theorem to control shrinking function brackets.

Suppose an integrable measurable-function class can be covered by finitely many function brackets of every positive $L^1(P)$ width. For an independent sample with common law $P$, the empirical measure satisfies ${\sup }_h|P_{n}h-Ph|\to 0$ on a common probability-one event. For one finite $\epsilon$-cover, the supremum is bounded by $\epsilon$ plus the largest empirical error among its endpoints. The strong law of large numbers makes that finite maximum vanish. Taking a countable sequence of widths decreasing to zero proves the assertion. A measurable supremum can be obtained from a pointwise separable function class; otherwise the probability-one-event formulation expresses the same pathwise conclusion.

If $\Theta$ is compact, $q(\theta ,x)$ is continuous in $\theta$ and measurable in $x$, and $\mathbb{E}{\sup }_{\Theta }|q(\theta ,X)|<\infty$, then an independent and identically distributed sample satisfies ${\sup }_{\Theta }|P_n{q}_{\theta }-P{q}_{\theta }|\to 0$ almost surely. Uniform continuity for each $x$, the dominated convergence theorem, and finite parameter nets produce arbitrarily narrow function brackets.

The empirical distribution function converges uniformly almost surely to the population distribution function.

For a cumulative distribution function $G$, let $Q(u)=\inf \{x:G(x)\ge u\}$, $0<u<1$. Right continuity gives $Q(u)\le x$ exactly when $u\le G(x)$, including at atoms. Thus applying $Q$ to independent uniform variables $U_i$ constructs an independent sample with distribution function $G$. Its empirical distribution function is $G_n(x)=F_n(G(x))$, where $F_n$ is the uniform-sample empirical distribution function. Consequently ${\sup }_x|G_n(x)-G(x)|\le {\sup }_{t\in [0,1]}|F_n(t)-t|$. For the uniform sample, monotonicity and a grid of mesh $1/m$ bound the last supremum by the largest error at grid points plus $1/m$. The strong law of large numbers on the countable union of all such grids proves the almost-sure limit. This construction proves the theorem for distributions with atoms as well as continuous distributions.