Past exam of the experimental design and methods course of the year 2 of the psychology (experimental) course of the University of Oxford / 2022 / 6 / b

Make the development process inspectable and separate exploratory decisions from independent confirmation. Create a preregistration for the validation stage before examining its outcome-relevant data, specifying recruitment, stopping rules, exclusions, missing-data treatment, item scoring, estimator, factor structure and criteria for evaluating it. The exploratory stage can also have a recorded protocol, including a planned parallel analysis rule; label data-driven item revisions and model changes as exploratory rather than disguising them as prior predictions. Document deviations and their reasons.^[ref]

Keep an immutable copy of the collected data, record cleaning steps, and provide a data dictionary, final item wording, scoring instructions and executable analysis code. Record software and package versions, random seeds for simulations or bootstrap calculations, and the steps needed to regenerate tables from the permitted data. These support computational reproducibility; an independently collected validation study or scientific replication addresses a different question.

Share appropriately de-identified data where consent and re-identification risk permit, with controlled access or a clear restriction statement where they do not. Protect lecturer and rater identities and disclose the pilot's convenience sampling and all item-selection steps. Report inconclusive and unfavorable findings as well as favorable ones. A registered report can obtain peer review of the validation plan before its results are known, reducing outcome-dependent publication incentives.^[ref]