Past exam of the digital social research: statistics core course of the University of Oxford / 2020 / 1 / Solution

Let ${\mu }_0$ and ${\mu }_1$ be the expected political efficacy scores in the two employment groups in the target population. The null hypothesis is $H_0:{\mu }_0-{\mu }_1=0$ and the two-sided alternative hypothesis is $H_1:{\mu }_0-{\mu }_1\ne 0$. A Welch's t-test compares the difference of sample means with its standard error, estimating the two group variances separately:The reference Student t-distribution has approximate, generally noninteger statistical degrees of freedom; it does not require equality of the two variances. Observations should be independent, with finite group variances; normal distributions give the usual small-sample rationale, while large samples permit an approximate normal sampling distribution under suitable conditions. A large sample cannot correct a biased sampling procedure or dependence between observations.^[ref]

A p-value is a probability calculated under the null hypothesis and the sampling assumptions: for this two-sided test it measures the probability of an absolute test statistic at least as large as the observed one. It is not the probability that the null hypothesis is true, the probability that chance caused the observed difference, or a measure of the effect size. If the significance level $\alpha =0.05$ was chosen in advance, the printed $p=0.049521$ falls just below it. This would reject $H_0$ at 5%, but not at 1%. The significance level describes the long-run probability of a type I error, rejecting a true null hypothesis; a type II error instead fails to reject a false null hypothesis. Statistical power is the rejection probability under a specified alternative hypothesis. None of these errors for this particular realised study can be read as a posterior probability from $p$.

Both the printed sample means and the positive confidence interval point toward greater political efficacy in group 0, the non-employed group. Their magnitudes disagree. Subtracting the printed sample means gives $8.656650-6.322812=2.333838$ scale units. Yet the printed 95% confidence interval has midpoint $(0.2822355+0.6743412)/2=0.47828835$, not $2.333838$. A two-sided Welch interval is centred on the difference of sample means. This violates the consistency identities for a two-sided t-test; the interval and estimates cannot describe the same reported comparison. If the interval alone were trusted, it would exclude zero and indicate an increase between about $0.28$ and $0.67$ units, but that is explicitly a conditional interpretation. A 95% confidence interval refers to repeated-sample coverage of a fixed statistical parameter, not a 95% posterior probability for that realised interval.^[ref]

There are additional arithmetic warnings. The printed $t=6.1616$ at $2022.4$ statistical degrees of freedom gives a two-sided p-value about $8.67\times 10^{-10}$, not $0.049521$. Thus taking the test statistic as authoritative would imply much stronger evidence, not merely a marginal 5% result. The employment counts sum to $2311$, but their percentages are $905/2311\approx 39.16\%$ and $1406/2311\approx 60.84\%$, not $39.27\%$ and $60.73\%$. If those counts also apply to the two political efficacy sample means, their weighted arithmetic mean would be about $7.237$, not the reported overall $8.348$; separate complete-case group sizes are not supplied, so that last cross-output comparison is conditional on a common sample. No uniquely corrected difference, standard error or interval follows from these contradictory fields.

The political efficacy summary describes a bounded observed scale from $0$ to $16$, with median $9$ and middle quartiles $7$ and $12$. The interquartile range is $5$; the overall sample mean $8.348$ being below the median does not, by itself, establish a particular distributional shape. These descriptive statistics cannot supply the missing coherent group variances or a standardized effect size. Even a corrected small p-value would establish an association under a statistical model, rather than a causal effect of employment. Age, other background covariates, selection bias and the quality of the political efficacy measurement remain relevant. A defensible report gives the conditional 5% decision and direction, identifies the contradictions, and withholds a single numerical effect-and-uncertainty claim until the output is checked.