For the normal linear model, the log-likelihood is
The full column rank of the design matrix makes invertible. Differentiating in gives the normal equations . With the minimized residual sum of squares substituted, differentiation in gives
These are the maximum-likelihood estimators almost surely; almost surely when and . The linear transformation of the multivariate normal distribution yields
Define the hat matrix . The fitted values are , the regression residuals are , and . The orthogonal projection has rank and annihilates . By Cochran's theorem,
Consequently , with bias . The unbiased estimator is
Its square root, the reported residual standard error, estimates ; the assertion of unbiasedness applies to the variance, not generally to its square root.
For the paper-strength analysis, let be the measured percentage and . Both normal linear models use independent errors of common variance . Their mean functions are for lm1, and for lm2, with separately fitted coefficients. The reported residual standard errors are and , respectively.
For lm1, the estimated conditional expectation at a new percentage is
The original data mean must be used for centering the new percentage; its numerical value is not supplied in the excerpt. Because , the two columns of this design matrix are orthogonal, and . Hence the coefficient standard errors in the output give
This estimates uncertainty in the mean. Predicting an individual future batch would additionally require the new-error variance, estimated by .
To compare the nested normal linear models, test against within the quadratic model. The Student t-test statistic is
Its two-sided p-value is . Equivalently is a partial F-test statistic with null distribution . Reject the linear restriction and prefer lm2 at the 5% level. Its residual spread is much smaller; the increase in the coefficient of determination from to supports the same conclusion, although the test is the relevant complexity-adjusted comparison.
For regression diagnostics, inspect regression residuals against fitted means and hardwood percentage for omitted curvature, a scale-location plot for nonconstant variance, and a quantile-quantile plot against the normal distribution for departures from the error assumption. Check unusual observations using regression leverage and Cook's distance, and examine residuals in collection or batch order if dependence is plausible. Independence and a common variance require substantive justification as well as these plots; a small p-value for a polynomial term does not itself check the error model.
Let be speed and the tool type. Each fit is a normal linear model with independent errors. The three mean specifications are
The constraints are treatment coding with type 1 as reference. The parameter counts are , respectively. Model 2 gives parallel lines; model 3 allows interaction terms and distinct slopes.
In the analysis of variance interaction row, adding three independent slope differences costs three degrees of freedom. Its extra sum of squares is the reduction in residual sum of squares, . Its mean square is , and its F-test statistic is
Thus the four missing entries are , , , and , to the precision allowed by the rounded output. This tests against the alternative that at least one slope difference is nonzero. Under and the normal linear model assumptions, the statistic has distribution . Its p-value gives no reason to reject at 5%.
The simpler common-line model is inadequate compared with the parallel-line model. Testing against at least one nonzero type effect, while retaining speed, gives the partial F-test
Its p-value is far below , so reject the common-line restriction. This reduction differs from the sequential type sum of squares in the displayed table, because that table adds type before speed. Recommend model 2: different intercepts and a common decreasing slope.
The selected coefficient estimates give
The intercept estimates the expected lifetime of type 1 at speed zero; if zero speed is outside the data range, it is only an extrapolated intercept. Its standard error is , and its Student t-test ratio is , with two-sided p-value . At any fixed speed, type 2 differs from type 1 by hours, with standard error , and p-value , giving little evidence of a difference. Types 3 and 4 exceed type 1 by and hours, with standard errors and ; their test ratios and and p-values and support positive differences. Each of these individual coefficient tests has null value zero, alternative nonzero, and null distribution .
The speed coefficient means a reduction of hours per extra revolution per minute, or hours per additional 100 rpm, for every type. Its standard error is ; the ratio has null distribution when the common slope is zero, and two-sided p-value .
The residual standard error estimates the common noise standard deviation in hours using 15 residual degrees of freedom, since . The coefficient of determination means that about of the corrected lifetime variation is explained by this fit. The overall F-test statistic tests the simultaneous restriction against at least one nonzero coefficient; its null distribution is and its p-value rejects an intercept-only mean. These conclusions remain conditional on suitable regression diagnostics for independent, homoscedastic, approximately normal errors.
The required sketch plots the four fitted lines. Its speed interval is illustrative because the observed speeds are not supplied; the ordering and vertical gaps are determined by the fitted coefficients.
Figure 1.
Parallel fitted tool-lifetime lines for the four tool types
.
Write and . With , the Gamma exponential dispersion family representation is
where
Substitution gives the original gamma density, so both unknown parameters are included. The exponential-family derivative identities yield and . Therefore
The exact natural-parameter canonical link function is . The usual inverse-link convention is , as used by the printed R fits; multiplying the link and coefficients by gives the natural-parameter convention. The sign convention changes neither the fitted means nor the weight matrix below.
A generalized linear model specifies independent responses in an exponential dispersion family, their means , and a link function connecting those means to a linear predictor. In the common notation,
where are known positive weights, is the common dispersion parameter, and is the variance function. The systematic part is
Here is a row of the known design matrix and contains the unknown regression coefficients. The link function is invertible on the permitted mean domain; the canonical link function takes the mean to the natural parameter. This separates distributional, predictor, and link assumptions: it does not require the response itself to be normally distributed or the mean itself to be linear in the covariates.
Both fits use the same Gamma distribution mean model and inverse link function. Their coefficient estimates, fitted means and hence their weight matrices coincide: the common dispersion parameter only multiplies the likelihood score by a scalar and therefore does not change its zero. The displayed calls likewise show the same family and formula; fixing dispersion in a summary changes the uncertainty calculation, not these coefficient estimates.
For and , the Fisher information weights are
Thus the common matrix is multiplied by dispersion in the exponential case and by in the fitted gamma case. Taking square roots of the diagonal covariances gives
For instance for the intercept, and the same factor applies to all the coefficients. The standard errors are smaller because the fitted dispersion parameter is below one, not because a different mean function was fitted.
For the gamma variance function, the Pearson dispersion estimator is
using residual degrees of freedom. Under a specified null dispersion , the scaled statistic based on squared Pearson residuals has an approximate distribution under the usual residual approximation.
Thus test1 corresponds to , equivalently gamma shape , the exponential distribution. test2 corresponds to , equivalently shape . The null hypotheses concern dispersion or shape, not whether the regression coefficients vanish.
The code computes lower-tail probabilities. Used as one-sided tests, the alternatives are and , respectively, equivalently shapes larger than one and three. At the 5% level the first null is rejected because the lower-tail probability is , while the second is not rejected because its probability is . If the intended alternatives are two-sided, , these displayed numbers must not be called two-sided p-values: doubling the smaller tail gives approximately and , with the same decisions. The chi-squared distribution approximation is not an exact finite-sample gamma identity.
The dispersion-one assumption is strongly contradicted by the preceding calculation. Use the gamma analysis with estimated dispersion, namely the F table. A failure to reject shape three does not establish that the true shape equals exactly three; retaining the estimated dispersion parameter is appropriate.
In the analysis of deviance for nested generalized linear models, a deviance reduction for extra coefficients is divided by when dispersion is estimated. The approximate null calibration is . For the type term, the null is that the two type contrasts vanish, against at least one nonzero contrast. The statistic is
with approximate null distribution and p-value . For position, the null is that its coefficient vanishes after accounting for type; the statistic is
with approximate null distribution and p-value . At 5% there is evidence that component type affects the expected failure time, and no significant additional position effect. The table is sequential: the type comparison is to the intercept-only model, while the position comparison adjusts for type. The individual gamma coefficient tests also suggest that type 3 accounts for the clearer type difference. The inverse link function means its positive contrast corresponds to a lower fitted mean failure time, holding position fixed.
The three Poisson regression models have independent counts and a logarithmic link function: their mean predictors include both pc and urban in m1, only urban in m2, and only pc in m3. Nested comparisons through the full model are preferable to a direct comparison of the two nonnested one-predictor models.
To remove urban from m1, test against . The likelihood-ratio test statistic is the deviance difference
with approximate null distribution . It is below , so do not reject at 5%. The corresponding Wald p-value is consistent with this decision. To remove pc from m1, test against a nonzero coefficient, giving with approximate null distribution ; reject decisively. The Akaike information criterion also prefers m3, whose value is smaller than and .
Among these three models choose m3, retaining building cover and omitting the urban indicator. This is a comparison within the stated Poisson family; its absolute fit must still be checked, as the next part does.
The chosen Poisson regression has fitted mean
For zero building cover, its fitted expected number of floods over the 25-year observation window is . Increasing the covered proportion by multiplies the fitted expected count by . Thus a ten-percentage-point increase, , multiplies the expected count by about , an increase of . A one-percentage-point increase multiplies it by about . The full-unit multiplier compares proportions differing by one, not percentages differing by one.
These are associations in the conditional expectation with the recorded final-year building cover. The regression coefficient is not automatically a causal effect of changing development, especially since the count covers 25 years while cover was measured at the end.
The residual deviance is on residual degrees of freedom. Under a suitable Poisson deviance goodness-of-fit test approximation it is compared with , whose supplied 95th percentile is . Since is much larger, the chosen Poisson model does not provide a satisfactory absolute fit, despite being preferred among the three candidates.
The ratio suggests substantial overdispersion or mean-model misspecification. It is a deviance ratio, not the exact Pearson dispersion estimator, which cannot be computed from the excerpt. Check residual patterns, nonlinear effects, unusual rivers and dependence. A Quasi-Poisson regression could adjust uncertainty under an estimated dispersion, and a negative binomial regression could model extra count variation; a more adequate mean model may also be needed. The nominal Poisson tests in part (a) should be interpreted in light of this failure, rather than treated as a validated final analysis.
A 100-year return level is a flow threshold whose probability of being exceeded in a given year is one percent, under a stationary model for annual extremes. Equivalently, its long-run average waiting time between years with an exceedance is one hundred years. It does not imply that exceedances occur regularly one hundred years apart, or that one is guaranteed in any particular hundred-year interval. This definition uses yearly exceedance events even when the underlying observations are daily maxima.
This is the generalized Pareto distribution for the excess , with scale and shape . Its support requires . The Pickands-Balkema-de Haan theorem says that, for distributions in an appropriate extreme-value domain of attraction, the conditional distribution of excesses above a sufficiently high threshold approaches a generalized Pareto form. This makes it the natural peaks-over-threshold method model; it is an asymptotic justification, not an assertion that every threshold is sufficiently high.
For , the tail decays as a power and has no finite upper endpoint, giving a heavy tail. For , the limiting distribution is exponential, with survival probability . For , there is a finite upper endpoint , giving a bounded tail. Thus the sign of the shape parameter distinguishes heavy, exponential-type and bounded tails.
In the original PDF's probability plot, most points lie close to the diagonal. The quantile plot likewise agrees well over most of the range, but the largest empirical flow, around , is well above its fitted quantile, around . The density plot captures the strongly decreasing right-skewed bulk of the excesses. The return level plot shows a broadly reasonable fit to most observations, with one conspicuously high extreme and increasingly wide uncertainty at long return periods. Consequently the generalized Pareto distribution is a reasonable first approximation, but its most extreme tail is less convincingly represented.
Reading the return-level curve at the 100-year tick gives
These are approximate graphical readings, not parameter-based calculations; rounding to whole units would give about with an interval roughly to . The two blue curves are uncertainty bounds for the fitted return level, not prediction bounds for individual observations. The daily-data independence assumption and the stability of the fit as the threshold changes should be checked, for example by examining clusters of high flows and using declustering of extremes where necessary. A 100-year level also extrapolates beyond the 25-year record.
For the geometric distribution supported on positive integers, set . The geometric series and its derivative give
Equivalently, the sum of the tail probabilities is . For , the expected wait is 50 years, so this is the 50-year return level. The exceedance probability determines the return period, but does not by itself determine a numerical flow rate.
Consider a competing interpolant and set . At each spline knot, . If its second derivative roughness penalty is infinite there is nothing to prove, so assume . Its absolutely continuous first derivative makes absolutely continuous as well, with defined almost everywhere.
On each knot interval the natural cubic spline has constant third derivative. Integrating by parts once gives
The last term vanishes because is zero at both endpoints. Summing over intervals cancels the interior boundary terms, since and are continuous at the knots. The exterior terms vanish because . Hence . Expanding the square now gives
Since , equality requires almost everywhere. Absolute continuity then makes affine. It has at least two distinct zeros because , so . The natural cubic spline is the unique minimizer. This proves the minimum roughness property with absolutely continuous first derivatives, covering the weaker regularity in the question rather than assuming competitors are twice continuously differentiable. The case of two knots is included: the minimizing spline is their straight-line interpolant.
For woman , write for the number of births, for the number with defects, for baseline age, and for exposure. Let be the common marginal defect probability for her births. The first fit is a grouped grouped-binomial logistic regression:
It assumes independent women and, conditionally on their covariates, independent births with the same probability within each woman. The supplied birth totals are the binomial denominators; fitting the proportions with weights is equivalent to this grouped-binomial specification. The fitted linear predictor is .
The second fit is a generalized additive model with the same logit link function and the mean specification
The functions are penalized cubic regression splines with natural boundary conditions. Centering constraints such as separate them from the intercept; the fitted centered intercept is . Their second derivative roughness penalties control complexity.
The call estimates a common scale rather than fixing it at one. Its working variance specification for the counts is
Equivalently the variance of is . This is the working moment interpretation of an overdispersed binomial generalized additive model: the code uses binomial deviance for fitting and estimated scale for inference. With , it is not an exact independent-binomial sampling model. Women are still treated as independent groups, but within-woman dependence or unobserved heterogeneity can motivate the extra dispersion. The quasibinomial regression interpretation states what the scaled analysis assumes without inventing a full probability distribution for its counts.
The first model's residual deviance is on residual degrees of freedom, a ratio of about . This is far above the scale expected from a well-fitting dispersion-one grouped binomial regression. It suggests an inadequate mean function, overdispersion, or both. The linear age and exposure restrictions may miss nonlinear associations; the generalized additive model allows those shapes to be estimated through penalized cubic regression splines rather than assumed.
The very small exposure Wald p-value in the first fit suggests an association, but does not establish that a linear logit effect is adequate. Nor does the nonsignificant linear age term exclude every possible nonlinear age effect. Allowing estimated scale also addresses the excessive residual variation, which can arise from clustered data because several births belong to the same woman. The second model's estimated scale confirms that permitting smooth means has not removed all extra variation. The reason to proceed is poor initial fit and possible nonlinear effects, with dispersion-aware inference, not merely the wish to obtain more significant tests. These observational fits alone do not establish causation by the disaster.
The trace of the smoother's influence matrix is its total effective degrees of freedom. Include the unpenalized intercept as well as both centered smooth terms:
The Ref.df entries are for approximate significance calibration and must not be summed to obtain this trace. A consistency check using the generalized cross-validation form and scale estimate gives ; the small difference is explained by rounding in the displayed quantities. The required trace is about .
The original PDF's age smooth is nearly flat, its uncertainty band includes zero across the age range, and its approximate p-value is . The exposure smooth is increasing and convex: it is fairly flat at low exposure and rises more steeply at high exposure. Its effective degrees of freedom suggest a modest departure from a straight line, rather than a highly oscillatory relationship.
A suitable simpler model is therefore the grouped quasibinomial regression
with age omitted and dispersion estimated. A positive quadratic coefficient captures the convex exposure relationship; retaining the linear term respects the usual polynomial hierarchy. This model should be refitted, rather than assigning coefficients from the smooth plot. Compare its residual pattern and dispersion-adjusted fit with the additive model to check that the quadratic approximation is adequate. An exposure-only quadratic logit model is a justified parsimonious candidate, while the graph does not justify a sharp threshold or a zero-dispersion model.
Introduce a latent indicator for a structural zero. In this parametrization , and conditional on the response has a Poisson distribution with mean . The zero-inflated Poisson regression uses
Its observed probability mass function is at zero and at a positive count. The zero-mixing probability here is explicitly ; it is distinct from the weight of the count component.
For the expectation-maximization algorithm, start with positive and . At iteration , the E-step computes
This distinguishes structural zeros from Poisson-generated zeros. Up to terms independent of the new parameters, the expected complete-data log-likelihood is
It separates into a fractional-response logistic fit and a weighted Poisson fit. Because treatment is binary, each component is saturated over its two treatment groups, so the M-step has closed forms.
For , let , , , and . The M-step score equations give
Here , since every positive count has . The explicit coefficient updates are
Iterate the E- and M-steps until the observed log-likelihood and parameter estimates stabilize. This is the EM algorithm for zero-inflated Poisson regression with explicit binary-group updates; merely naming two regression routines would not supply these expressions. Both treatment groups must be represented for both contrasts to be identifiable. Zero fitted group means or endpoint mixing probabilities are boundary solutions, interpreted through limits of the log or logit coefficients. As with other mixture models, multiple starts help distinguish competing stationary solutions; EM increases the likelihood but does not guarantee a global maximum from an arbitrary start.
There are two different effect scales in this zero-inflated Poisson regression. The count component describes the mean conditional on belonging to the susceptible component, including its possible zero outcomes. Its logarithmic link function coefficients exponentiate to ratios of those conditional means. The zero component describes the structural-zero probability ; its logit coefficients exponentiate to odds ratios, not probability ratios.
Holding the other covariates fixed, Centre B versus Centre A changes the susceptible count mean by the factor
an increase of about . It simultaneously multiplies the odds of being a structural zero by
Thus Centre B is associated both with higher count intensity among susceptible patients and with greater odds of belonging to the never-at-risk class.
Use no personality disorder as the reference. For women, borderline disorder multiplies the susceptible count mean by , and other disorder multiplies it by . Because the count model has sex-by-disorder interaction terms, the corresponding male comparisons must add the interactions before exponentiating:
These compare men with the stated disorder to otherwise comparable men without a disorder; the female comparisons use female reference patients. The interaction multipliers and alone are ratios of these sex-specific disorder ratios, not the overall male disorder effects.
In the zero component there is no sex-by-disorder interaction. Borderline and other disorders multiply the structural-zero odds relative to no disorder by and , respectively, holding centre fixed; these comparisons are the same for both sexes in the fitted zero model.
The marginal mean is . Therefore none of these count multipliers alone is the population-average change in expected episodes when the same covariate also changes . For a count coefficient change and zero-logit change from baseline zero predictor , the marginal mean ratio is
For centre use ; for disorder use its sex-specific count contrast and its zero contrast. This is the component and marginal effects in zero-inflated regression distinction needed to interpret these results carefully.
Let mean membership in the structural-zero component. Under the fitted zero-inflated Poisson regression, , whereas . By Bayes theorem, the posterior structural-zero probability is
Using the printed predictions for this patient gives
The fitted conditional probability is about . It is larger than the prior fitted structural-zero probability , because observing no episodes increases the probability of latent membership in that component. The interpretation “never at risk” is the model's structural class; an observed six-month zero alone does not identify the class with certainty.

Articles by others on the same topic (0)

There are currently no matching articles.