Past exam of the experimental design and methods course of the year 2 of the psychology (experimental) course of the University of Oxford / 2022 / 8

Bootstrapping estimates sampling uncertainty by repeating an analysis on resampled data. In the ordinary nonparametric bootstrap, draw observations with replacement from the empirical sample, retaining its size, and recalculate the statistic many times. The resulting distribution can estimate a standard error, assess bias or form a justified bootstrap confidence interval. It is useful when an analytic sampling distribution is difficult to obtain.

The resampling unit must match the design: keep paired measurements together, and resample independent clusters when observations are clustered. Resampling individual ratings as if they were independent lecturers would misrepresent uncertainty. The ordinary bootstrap assumes the sample and chosen resampling scheme adequately represent the relevant data-generating variation; it does not repair selection bias, guarantee accurate inference from a tiny sample, or succeed for every statistic. Specify the interval method and computational settings, since a simple percentile interval is not universally appropriate.

Preregistration creates a time-stamped record of research questions and planned methods before the relevant results are examined. Specify the design, recruitment and stopping rules, outcome definitions, exclusions, principal tests, model choices and any conditional decision rules. For existing data, disclose prior access and what was already known. A sufficiently precise record helps readers distinguish prior predictions from choices prompted by the results.^[ref]

It does not prohibit useful changes or new discoveries: report departures, their timing and reasons, and distinguish exploratory analysis from confirmatory analysis. A vague or selectively followed registration does not solve selective reporting, and preregistration cannot repair a poor measure or inadequate design of experiments. A registered report adds pre-result peer review and conditional publication acceptance, which are not implied by registration alone.

HARKing means presenting a hypothesis generated after inspecting results as though it had been specified beforehand. For example, after trying several possible factor structures in the lecturer data, a researcher might claim that the best-fitting structure was their original prediction. The problem is the misrepresented evidential history, not the act of discovering a pattern.

Such undisclosed selection makes apparent confirmation too persuasive because the reported test does not represent the whole search that produced the hypothesis. Label new ideas as exploratory, record prior plans through preregistration, disclose the search and evaluate the resulting prediction in independent data. Registered reports and transparent reporting can reduce incentives for that misrepresentation while preserving legitimate exploratory work.^[ref]