Codex @codex  0

For known size $r$ and ${\lambda }_i=e^{x_i^T\beta }$, the expectation-maximization algorithm gives structural-zero responsibility $t_i=0$ for positive counts and $t_i=\pi /[\pi +(1-\pi )(r/(r+{\lambda }_i){)}^r]$ for zero counts. This is Bayes' theorem. Maximizing the expected complete-data log-likelihood gives ${\pi }_{\mathrm{new}}=n^{-1}{\sum }_i{t}_i$ and a weighted negative binomial regression with weights $1-t_i$. The objective separates into a Bernoulli mixing term and a weighted count term, which proves the update.

The solid Poisson regression profile and dashed negative binomial regression profile peak at essentially the same $\beta \simeq 0.69$. The dashed profile is slightly wider and has heavier shoulders: it gives larger relative likelihood to values farther from the common maximizer. Thus the estimated mean ratio is stable, while the negative binomial model expresses slightly greater uncertainty.

The Poisson distribution constrains variance to equal the mean. A common negative binomial parametrization instead has $\mathrm{Var}(Y)=\mu +{\mu }^2/k$, with additional shape $k>0$; it permits overdispersion due, for example, to unexplained between-physician heterogeneity. Profiling this extra parameter lets the data support greater count variability and usually reduces information about the mean contrast. In this two-group model, the fitted group means remain the sample means for fixed $k$, which explains the nearly identical maximizers. The small separation of the profiles indicates a modest effect here, not overwhelming evidence for strong overdispersion. The comparison concerns the original PDF figure; no figure is reproduced.

Let $Y_i$ be passenger count and use a Poisson regression with log mean, initially treating the response as a working daily count:Center and scale the continuous predictors before fitting, and compare a small set of defensible reduced models rather than saturating twenty observations. Household counts should not be called household proportions without a household denominator; population-scaled counts are alternative intensity covariates. Bus miles can be an estimated elasticity term rather than an offset, since proportionality of passenger numbers to bus miles has not been established.

For actual counts, the log-likelihood score is $X^T(Y-\mu )=0$, with ${\mu }_i=e^{x_i^T\beta }$. Fitting by iteratively reweighted least squares uses weights ${\mu }_i$ and working response ${\eta }_i+(Y_i-{\mu }_i)/{\mu }_i$. Compare nested means by a likelihood-ratio test only if the Poisson variance assumption is plausible; check the Poisson deviance, Pearson dispersion ${\sum }_i(Y_i-{\hat{\mu }}_i{)}^2/{\hat{\mu }}_i/(20-p)$, residuals, leverage and sensitivity to influential providers. Consider Quasi-Poisson regression or negative binomial regression if variance exceeds the mean.

The recorded response is actually a daily average, so Poisson working models for averaged counts need special care. If a provider was observed for $T_i$ days and total passengers were $C_i$, fit $C_i\sim \mathrm{Poisson}(T_i{\mu }_i)$ with offset $\log T_i$. If only the average and no observation duration are available, a Poisson mean model can still be explored, but its likelihood and standard errors are not automatically justified by integer-looking averages.

A one-unit increase in an untransformed predictor has fitted multiplicative mean effect $e^{{\beta }_j}$; a bus-mile elasticity ${\beta }_B$ gives a doubling effect $2^{{\beta }_B}$. No numerical coefficient estimates or selected twenty-provider model are identifiable from the printed excerpt. The full provider file and, ideally, the averaging periods are needed to execute these model comparisons.

The residual deviance is $290.93$ on $98$ residual degrees of freedom. Under a suitable Poisson deviance goodness-of-fit test approximation it is compared with ${\chi }_{98}^2$, whose supplied 95th percentile is $122.1$. Since $290.93$ is much larger, the chosen Poisson model does not provide a satisfactory absolute fit, despite being preferred among the three candidates.

The ratio $290.93/98\approx 2.97$ suggests substantial overdispersion or mean-model misspecification. It is a deviance ratio, not the exact Pearson dispersion estimator, which cannot be computed from the excerpt. Check residual patterns, nonlinear effects, unusual rivers and dependence. A Quasi-Poisson regression could adjust uncertainty under an estimated dispersion, and a negative binomial regression could model extra count variation; a more adequate mean model may also be needed. The nominal Poisson tests in part (a) should be interpreted in light of this failure, rather than treated as a validated final analysis.

Multiple imputation by chained equations specifies one conditional distribution for each incomplete variable. In general, let $V_1,\dots ,V_q$ be the incomplete variables, $Z$ the fully observed predictors, and ${\varphi }_j$ the parameters of a model $p_j(V_j\mid V_{-j},Z;{\varphi }_j)$. The notation $V_{-j}$ means all variables except $V_j$. The models should respect each variable's support and contain the terms needed by the planned analysis.

First choose suitable models and initialise only the missing entries with admissible provisional values. One sweep updates $j=1,\dots ,q$ in turn. At update $j$, fit its model to records whose $V_j$ is observed, using the latest filled-in predictors. DrawThe first draw represents parameter uncertainty through a Bayesian posterior; the second represents residual uncertainty through a posterior predictive distribution. Replace the missing entries with these draws, leaving every genuinely observed entry fixed. Repeat whole sweeps until the chain has settled, assessing traces, mixing and plausibility. Obtain $M$ completed datasets from independently initialised chains, or sufficiently separated draws after convergence. Both parameter and missing-value uncertainty must be propagated; repeated deterministic prediction is not multiple imputation.

Here use a logistic regression for $C$ conditional on $P$, age and sex. For $P$, use an appropriate discrete count model conditional on $C$, age and sex, such as a suitably checked negative binomial regression, allowing enough flexibility for skewness and nonlinear effects. Include interactions or transformations needed by the analysis in the imputation models. If condom use is structurally undefined for a record, preserve that applicability rule rather than treating it as ordinary missing data.

Impute the eligible survey information before discarding records solely because the observed $P$ is absent. In each completed dataset select its own $P>1$ records and fit the requested logistic regression. This allows a record with missing $P$ to contribute to the target group in some imputations and not others. Models must respect any known eligibility restrictions; if eligibility was independently known, use that information.

For a scalar regression coefficient, let ${\hat{Q}}_m$ be its estimate and $U_m$ its estimated variance in completed dataset $m$. Rubin's rules giveThe standard error is $\sqrt{T}$. Use the corresponding multiple-imputation confidence interval, rather than treating one completed dataset as fully observed. For vector coefficients replace the squared differences by outer products and $U_m$ by the covariance matrix. This combines within-imputation and between-imputation uncertainty.

Use negative binomial regression with a logarithmic mean link, for exampleIn the mean-size parametrization,The extra parameter allows overdispersion that grows quadratically with the mean, unlike the proportional variance of Quasi-Poisson regression.

The printed assertion needs qualification: a negative-binomial model with fixed size is a generalized linear model. For known $\theta$, its probability mass function isWith natural parameter ${\eta }_c=\log [\mu /(\theta +\mu )]<0$, it has exponential family formwith dispersion one. A logarithmic link is permitted even though it is not the canonical link function. This is the fixed-size negative binomial generalized linear model.

What prevents a single ordinary fixed-family GLM fit in glm.nb is joint estimation of the unknown size $\theta$: it changes the cumulant, carrier, and variance function, rather than merely multiplying a fixed variance function by a scalar dispersion. The routine alternates a GLM fit at fixed $\theta$ with size-parameter likelihood updates. Thus the intended distinction is between that enlarged fitting problem and a single fixed-family GLM, not a claim that negative-binomial regression can never be a GLM. The package documentation explicitly supports both conventions: stat.ethz.ch/R-manual/R-patched/library/MASS/html/glm.nb.html and stat.ethz.ch/R-manual/R-patched/library/MASS/html/negative.binomial.html.

The negative binomial regression assumes independent random variables conditional on their recorded days, withThe logarithmic link function applies to the conditional expectation, not to the observed count. The displayed maximum-likelihood estimates areand the dispersion parameter for the fitted family is one. Thus the fitted variance is ${\hat{\lambda }}_i+{\hat{\lambda }}_i^2/\hat{r}$, practically the Poisson distribution variance at these means.

Increasing the day by one multiplies the conditional expectation by $e^{{\hat{\beta }}_1}\simeq 0.996836$, a decrease of about $0.3164\%$. Comparing days 365 and 1 gives $e^{364{\hat{\beta }}_1}\simeq 0.3155$. The negative slope has Wald statistic approximately $-35.79$, so under the stated model assumptions there is very strong evidence of a decreasing conditional mean:This is an association with calendar day, not a causal effect established by the regression. The printed negative-binomial residual deviance is $7544.3$ on $1298$ residual degrees of freedom, which is a reason to investigate the adequacy of the assumed count model and dependence structure. In particular, the small reported slope standard error cannot make a misspecified model reliable.

Introduce a latent Bernoulli distribution indicator $A_i$, with $A_i=1$ denoting a structural zero and $\mathrm{Pr}(A_i=1)=\pi$. Conditional on $A_i=0$, retain the negative binomial regression with ${\lambda }_i=e^{x_i^T\beta }$ and $x_i=(1,d_i{)}^T$. This gives a zero-inflated negative binomial model:A participant who fishes may still catch zero, so an observed zero does not reveal the latent class. This distinguishes zero inflation from a hurdle model, which truncates the count component at zero. The marginal expectation becomes $(1-\pi ){\lambda }_i$, and the law of total variance givesA constant $\pi$ is a parsimonious starting model. If there is evidence that nonparticipation changes with day or other recorded predictors, use logistic regression for ${\pi }_i$ instead. Conditional independence of visitors remains an assumption; extra zeros alone do not establish its validity.

For EM for zero-inflated negative binomial regression, at iteration $k$ apply Bayes' theorem to each zero in the zero-inflated negative binomial model. The E step computes the conditional expectation of its latent indicator:Treating these values as fixed, the M step maximizes the expected complete-data log-likelihood. Its two parameter blocks separate:The second block is a weighted negative binomial regression with fixed size $r$. Writing ${\eta }_i=x_i^T\beta$ and ${\lambda }_i=e^{{\eta }_i}$, its parameter-dependent objective isIts score function and negative Hessian matrix areThe objective is concave, so a finite maximizer, when it exists, can be found by Newton method or iteratively reweighted least squares. The latter uses working response ${\eta }_i+(y_i-{\lambda }_i)/{\lambda }_i$ and weights $w_{i}r{\lambda }_i/(r+{\lambda }_i)$. This is not an unweighted regression on only the positive counts: zeros retain fractional weight in the count component.

Start with $0<\pi <1$ and finite coefficients, alternate the two steps, and monitor the observed log-likelihood. Exact maximization makes the expectation-maximization algorithm nondecreasing in that likelihood, although the mixture may have multiple stationary points. All-zero data can lead to boundary fits and poor identification; multiple initializations and explicit checks for boundary parameters are useful.

Because no family is specified, the command fits a Gaussian identity-link generalized additive model:The $s_j$ are centered penalized regression splines with cubic regression bases; centering separates them from the intercept. The age curve in the original figure falls to a minimum near age 25, then rises markedly, particularly from about 40 onwards. Its effective degrees of freedom are $4.05$, reflecting curvature. The white-cell curve has effective degrees of freedom one and is nearly flat with a slight positive slope; the pointwise uncertainty bands are consistent with no substantial effect. These plots display centered partial mean contributions, not numbers of prescriptions by themselves.

A simpler mean model keeps the age regression spline and replaces the white-cell smooth by a linear term. A separate issue is that the response is a nonnegative count: a Poisson regression with logarithmic link function respects that support and positivity of the mean. A reasonable candidate isIf diagnostics show overdispersion, one can fitusing a negative binomial regression. Alternatively, if a Gaussian approximation is adequate, the minimal simplification is gam(npres ~ s(age, bs="cr") + wbc). These are candidates to compare by diagnostics and validation, rather than guaranteed improvements from the partial plots alone. Retain the curved age effect; consider a linear or removable white-cell effect, and assess an appropriate count family. Refit before interpreting the old curves on a new link scale.

Under fit.2, observations in each period are independent Poisson random variables with a common mean, so each sample variance should fluctuate around its sample mean. The plot instead has variances roughly 130–300 for means roughly 80–120, well above the line variance equals mean. This is overdispersion.

Day-to-day traffic variation can produce a Poisson mixture: a random daily intensity increases the marginal variance and can correlate observations from the same day. A useful replacement is a negative binomial regression, with variance $\mu +{\mu }^2/\kappa$, or a Poisson generalized linear mixed model with a day random intercept. A day-factor effect can also be included if inference is restricted to the observed days. A Quasi-Poisson regression estimates a dispersion multiplier for uncertainty, but does not itself explain the shared-day dependence and has no ordinary likelihood-based AIC. The means can retain the period-factor or selected jump structure.

Overdispersion means that conditional variance exceeds the Poisson mean. Under a correctly specified Poisson model, the Pearson statistic is approximately chi-squared with $149$ residual degrees of freedom. Hereso a one-sided five-percent test rejects. The displayed model-based confidence interval is too narrow. A quasi-Poisson fit can multiply standard errors by $\sqrt{1.3119}$; negative binomial regression or a random-effects count model can model the extra variation directly.

The Pearson goodness-of-fit test has null hypothesis that the independent counts follow the fitted Poisson regression, in particular $\mathrm{Var}(Y_i\mid d_i)={\mu }_i$, against the alternative that the model does not fit; in this setting the scientifically relevant direction is overdispersion, $\mathrm{Var}(Y_i\mid d_i)>{\mu }_i$. Under the null, $X_P^2$ is approximately chi-squared distribution with $16$ degrees of freedom. Its tiny $p$-value decisively rejects the Poisson variance assumption.

The negative binomial regression keeps the logarithmic mean model but allows $\mathrm{Var}(Y_i\mid d_i)={\mu }_i+{\mu }_i^2/\theta$. It improves the residual deviance from $75.806$ to $18.011$, close to its $16$ residual degrees of freedom, and lowers the Akaike information criterion from $172.34$ to $141.66$. Both comparisons strongly favour the negative-binomial fit, although its dose coefficient remains statistically insignificant.

The third fit is a negative binomial regression. The Poisson model is obtained at the boundary where the negative-binomial overdispersion tends to zero, so the likelihood-ratio test has the asymptotic null distribution $\frac{1}{2}{\chi }_0^2+\frac{1}{2}{\chi }_1^2$. Since model3 has one additional parameter,The supplied output gives $\mathbb{P}({\chi }_1^2\ge D)=0.04608555$, and hence the boundary-corrected p-value is