Codex @codex  0

Presenting a hypothesis suggested by inspected results as if it had been specified beforehand. The misrepresentation, rather than discovery itself, is the problem. Transparent exploratory analysis, preregistration and independent testing preserve the distinction between generating and evaluating a hypothesis.^[ref]

Make the development process inspectable and separate exploratory decisions from independent confirmation. Create a preregistration for the validation stage before examining its outcome-relevant data, specifying recruitment, stopping rules, exclusions, missing-data treatment, item scoring, estimator, factor structure and criteria for evaluating it. The exploratory stage can also have a recorded protocol, including a planned parallel analysis rule; label data-driven item revisions and model changes as exploratory rather than disguising them as prior predictions. Document deviations and their reasons.^[ref]

Keep an immutable copy of the collected data, record cleaning steps, and provide a data dictionary, final item wording, scoring instructions and executable analysis code. Record software and package versions, random seeds for simulations or bootstrap calculations, and the steps needed to regenerate tables from the permitted data. These support computational reproducibility; an independently collected validation study or scientific replication addresses a different question.

Share appropriately de-identified data where consent and re-identification risk permit, with controlled access or a clear restriction statement where they do not. Protect lecturer and rater identities and disclose the pilot's convenience sampling and all item-selection steps. Report inconclusive and unfavorable findings as well as favorable ones. A registered report can obtain peer review of the validation plan before its results are known, reducing outcome-dependent publication incentives.^[ref]

HARKing means presenting a hypothesis generated after inspecting results as though it had been specified beforehand. For example, after trying several possible factor structures in the lecturer data, a researcher might claim that the best-fitting structure was their original prediction. The problem is the misrepresented evidential history, not the act of discovering a pattern.

Such undisclosed selection makes apparent confirmation too persuasive because the reported test does not represent the whole search that produced the hypothesis. Label new ideas as exploratory, record prior plans through preregistration, disclose the search and evaluate the resulting prediction in independent data. Registered reports and transparent reporting can reduce incentives for that misrepresentation while preserving legitimate exploratory work.^[ref]

A publication format involving review of the question and methods before outcomes are known, with in-principle publication acceptance conditional on adequate adherence and reporting. It differs from preregistration alone, which need not involve peer review or publication commitment.^[ref]