= Solution
\b[Make the development process inspectable and separate exploratory decisions from independent confirmation.] Create a <preregistration> for the validation stage before examining its outcome-relevant data, specifying recruitment, stopping rules, exclusions, missing-data treatment, item scoring, estimator, factor structure and criteria for evaluating it. The exploratory stage can also have a recorded protocol, including a planned <parallel analysis> rule; label data-driven item revisions and model changes as exploratory rather than disguising them as prior predictions. Document deviations and their reasons.<Preregistration by Center for Open Science>[What is Preregistration?; planned versus unplanned work; FAQ on detail and prior data knowledge]{ref}
Keep an immutable copy of the collected data, record cleaning steps, and provide a data dictionary, final item wording, scoring instructions and executable analysis code. Record software and package versions, random seeds for simulations or <bootstrap> calculations, and the steps needed to regenerate tables from the permitted data. These support <computational reproducibility>; an independently collected validation study or <scientific replication> addresses a different question.
Share appropriately de-identified data where consent and re-identification risk permit, with controlled access or a clear restriction statement where they do not. Protect lecturer and rater identities and disclose the pilot's <convenience sampling> and all item-selection steps. Report inconclusive and unfavorable findings as well as favorable ones. A <registered report> can obtain peer review of the validation plan before its results are known, reducing outcome-dependent publication incentives.<Registered Reports by Center for Open Science>[Details and Workflow: Stage 1 review, in-principle acceptance, Stage 2 reporting]{ref}
Back to article page
