Past exam of the experimental design and methods course of the year 2 of the psychology (experimental) course of the University of Oxford / 2022 / 6

Treat the forty-person convenience sample as a pilot, then develop and validate the measure in its intended population. Start by defining whose organizational ability the questionnaire measures and who answers it: lecturer self-reports and student ratings are different measurements. Specify the relevant activities and context. Discuss the candidate items with lecturers and appropriate users, seek expert review, and use cognitive interviewing to find ambiguous wording, double-barreled items and irrelevant content. This establishes content validity rather than assuming that correlations alone identify the intended construct.

Recruit a larger, better-targeted sample covering relevant roles, disciplines and institutions. A convenience sample of adults who cannot meaningfully judge lecturers' organization is not made suitable simply by increasing its size. Plan the sample size using plausible factor loadings, communalities, number of dimensions and indicators, response distributions, missingness and required estimation stability, preferably with simulation. Forty observations for sixteen items offer limited grounds for a stable generalizable structure; no universal participants-per-item cutoff makes this analysis automatically impossible or automatically adequate. If students rate the same lecturers, recognize that the ratings are clustered rather than independent lecturer-level observations.

Check missing responses, item distributions, scoring direction and correlations, identifying redundant or poorly functioning items. If the underlying dimensions are not already well specified, use exploratory factor analysis to model shared variation among items. Principal component analysis instead forms combinations explaining total observed variation; it is not the same latent measurement model. Choose a suitable common-factor estimator, such as principal axis factoring, or maximum likelihood estimation when its assumptions are credible. For ordered response categories, consider an ordinal estimator and polychoric correlations, while checking whether category counts support their estimation. Do not treat those correlations as assumption-free.^[ref]

Choose the number of factors using parallel analysis, the scree plot, substantive interpretation and the quality of the resulting model, rather than an eigenvalue-one rule alone. Parallel analysis compares the observed eigenvalues with a reference generated from uncorrelated data of comparable dimensions; its implementation must match the chosen correlation and extraction approach.^[ref] Use oblique factor rotation if organizational dimensions can plausibly correlate. Inspect factor loadings, cross-loadings, communalities, factor correlations and model discrepancies, and retain enough substantively relevant indicators for each dimension. Do not repeatedly delete items solely to improve numerical fit at the expense of coverage.

After developing the structure, test it using confirmatory factor analysis in an independently recruited validation sample, with the proposed structure fixed beforehand. Splitting the existing forty participants would not solve the shortage of information. Evaluate model fit and uncertainty, measurement reliability, test-retest reliability where stability is intended, and evidence for construct validity using related, distinct and relevant behavioral measures. Check measurement invariance before comparing scores across groups. Report alternative plausible models, all item decisions, sampling limitations and the distinction between an initial exploratory instrument and an independently validated measure.

Make the development process inspectable and separate exploratory decisions from independent confirmation. Create a preregistration for the validation stage before examining its outcome-relevant data, specifying recruitment, stopping rules, exclusions, missing-data treatment, item scoring, estimator, factor structure and criteria for evaluating it. The exploratory stage can also have a recorded protocol, including a planned parallel analysis rule; label data-driven item revisions and model changes as exploratory rather than disguising them as prior predictions. Document deviations and their reasons.^[ref]

Keep an immutable copy of the collected data, record cleaning steps, and provide a data dictionary, final item wording, scoring instructions and executable analysis code. Record software and package versions, random seeds for simulations or bootstrap calculations, and the steps needed to regenerate tables from the permitted data. These support computational reproducibility; an independently collected validation study or scientific replication addresses a different question.

Share appropriately de-identified data where consent and re-identification risk permit, with controlled access or a clear restriction statement where they do not. Protect lecturer and rater identities and disclose the pilot's convenience sampling and all item-selection steps. Report inconclusive and unfavorable findings as well as favorable ones. A registered report can obtain peer review of the validation plan before its results are known, reducing outcome-dependent publication incentives.^[ref]