Past exam of the experimental design and methods course of the year 2 of the psychology (experimental) course of the University of Oxford / 2022 / Section B

TODO: the supplementary model call and tables are needed to identify the actual analyses. The described categorical predictors are compatible with a multiple linear regression containing a distance factor, a size factor and possibly their interaction; a factorial analysis of variance can represent the same mean structure. The mention of a coefficient table does not establish the exact fitted terms, predictor coding, estimation method or dependence assumptions. No particular fitted model is asserted.

TODO: the model's explained-variance value is absent. For an ordinary least-squares linear regression with an intercept, the coefficient of determination is $R^2=1-S{S}_{\mathrm{residual}}/S{S}_{\mathrm{total}}$. Its value multiplied by one hundred is the corresponding percentage of sample variation explained by that fitted model. Read the actual reported $R^2$ and distinguish it from the adjusted coefficient of determination; neither is a verified prediction-performance estimate on new data. Without the missing summary or sums of squares, no numerical percentage can be recovered.

TODO: the coefficients and their numerical $t$ and $p$ entries cannot be recovered. The coefficient estimate gives a modeled difference relative to the stated reference coding; its standard error quantifies uncertainty, and the usual coefficient test uses $t=\hat{\beta }/\mathrm{SE}(\hat{\beta })$ for a zero-coefficient null hypothesis. The p-value requires the appropriate reference distribution and statistical degrees of freedom. In a conventional R coefficient table an Estimate is usually an unstandardized regression coefficient, not automatically a standardized regression coefficient; the table and model documentation must establish which meaning of $\beta$ is intended. These definitions supply no missing numerical values.

TODO: directions, magnitudes and inferential conclusions require the unavailable coefficient table, coding and model specification. For illustration of interpretation only, let $D$ and $S$ be binary indicators whose reference levels are explicitly defined. If an ordinary linear model includes both factors and their interaction, its conditional mean has the formThen the distance difference is ${\beta }_D$ at the reference size and ${\beta }_D+{\beta }_{DS}$ at the other size; the interaction is the difference between those differences. If the interaction is omitted, the model assumes a common distance difference across sizes. These are conditional model meanings, not a claim that the missing analysis used this model or that either predictor had an observed effect. The uncertainty of a coefficient sum also involves their covariance, not merely their separate p-values.

TODO: no definite follow-up decision can be made without the fitted model, uncertainty and diagnostics. An included interaction can be clarified with simple-effects analysis, fitted cell summaries and confidence intervals on scientifically relevant differences. If the study aims to test whether the predictors combine additively but only an additive model was fitted, an adequately supported, prespecified interaction comparison may be relevant. Respect multiple testing when several comparisons are examined.

Check model fit, influential observations and heteroscedasticity; a response bounded between zero and one thousand may require particular care near its bounds, but that scale alone does not choose a replacement model. If observations are dependent, the analysis must account for that structure. Such checks are possible recommendations; the missing supplementary sheet prevents identifying which problems, if any, actually occurred.

TODO: an empirical Results section requires the missing output and cannot be supplied honestly here. A completed report would name the actual analysis, define the predictor reference levels and outcome units, summarize the relevant cells with sample counts and uncertainty, give overall model-fit information and the actual $R^2$ or other appropriate fit measure, and report each fitted coefficient with its standard error or confidence interval, test statistic and p-value. If an interaction was fitted, interpret conditional differences rather than treating reference-level coefficients as unconditional effects. State relevant regression diagnostics, departures from the planned analysis and any controlled follow-up comparisons. No observed coefficient, significance result or predicted value is inferred from the verbal scenario.

Treat the forty-person convenience sample as a pilot, then develop and validate the measure in its intended population. Start by defining whose organizational ability the questionnaire measures and who answers it: lecturer self-reports and student ratings are different measurements. Specify the relevant activities and context. Discuss the candidate items with lecturers and appropriate users, seek expert review, and use cognitive interviewing to find ambiguous wording, double-barreled items and irrelevant content. This establishes content validity rather than assuming that correlations alone identify the intended construct.

Recruit a larger, better-targeted sample covering relevant roles, disciplines and institutions. A convenience sample of adults who cannot meaningfully judge lecturers' organization is not made suitable simply by increasing its size. Plan the sample size using plausible factor loadings, communalities, number of dimensions and indicators, response distributions, missingness and required estimation stability, preferably with simulation. Forty observations for sixteen items offer limited grounds for a stable generalizable structure; no universal participants-per-item cutoff makes this analysis automatically impossible or automatically adequate. If students rate the same lecturers, recognize that the ratings are clustered rather than independent lecturer-level observations.

Check missing responses, item distributions, scoring direction and correlations, identifying redundant or poorly functioning items. If the underlying dimensions are not already well specified, use exploratory factor analysis to model shared variation among items. Principal component analysis instead forms combinations explaining total observed variation; it is not the same latent measurement model. Choose a suitable common-factor estimator, such as principal axis factoring, or maximum likelihood estimation when its assumptions are credible. For ordered response categories, consider an ordinal estimator and polychoric correlations, while checking whether category counts support their estimation. Do not treat those correlations as assumption-free.^[ref]

Choose the number of factors using parallel analysis, the scree plot, substantive interpretation and the quality of the resulting model, rather than an eigenvalue-one rule alone. Parallel analysis compares the observed eigenvalues with a reference generated from uncorrelated data of comparable dimensions; its implementation must match the chosen correlation and extraction approach.^[ref] Use oblique factor rotation if organizational dimensions can plausibly correlate. Inspect factor loadings, cross-loadings, communalities, factor correlations and model discrepancies, and retain enough substantively relevant indicators for each dimension. Do not repeatedly delete items solely to improve numerical fit at the expense of coverage.

After developing the structure, test it using confirmatory factor analysis in an independently recruited validation sample, with the proposed structure fixed beforehand. Splitting the existing forty participants would not solve the shortage of information. Evaluate model fit and uncertainty, measurement reliability, test-retest reliability where stability is intended, and evidence for construct validity using related, distinct and relevant behavioral measures. Check measurement invariance before comparing scores across groups. Report alternative plausible models, all item decisions, sampling limitations and the distinction between an initial exploratory instrument and an independently validated measure.

Make the development process inspectable and separate exploratory decisions from independent confirmation. Create a preregistration for the validation stage before examining its outcome-relevant data, specifying recruitment, stopping rules, exclusions, missing-data treatment, item scoring, estimator, factor structure and criteria for evaluating it. The exploratory stage can also have a recorded protocol, including a planned parallel analysis rule; label data-driven item revisions and model changes as exploratory rather than disguising them as prior predictions. Document deviations and their reasons.^[ref]

Keep an immutable copy of the collected data, record cleaning steps, and provide a data dictionary, final item wording, scoring instructions and executable analysis code. Record software and package versions, random seeds for simulations or bootstrap calculations, and the steps needed to regenerate tables from the permitted data. These support computational reproducibility; an independently collected validation study or scientific replication addresses a different question.

Share appropriately de-identified data where consent and re-identification risk permit, with controlled access or a clear restriction statement where they do not. Protect lecturer and rater identities and disclose the pilot's convenience sampling and all item-selection steps. Report inconclusive and unfavorable findings as well as favorable ones. A registered report can obtain peer review of the validation plan before its results are known, reducing outcome-dependent publication incentives.^[ref]

Use cluster analysis to discover groups of observations with similar profiles when no correct group labels have been supplied. For example, it could group lecturers by patterns of reported organizational behavior. Define the analysis unit and features, handle missing data, and choose scaling and a meaningful similarity measure; otherwise a feature with large numerical units may dominate the groups.

K-means clustering partitions observations around centers, whereas agglomerative hierarchical clustering produces nested groupings represented by a dendrogram. Their results include group membership and interpretable profiles, conditional on these choices. Check stability across resampling and reasonable analysis choices and, when possible, in new data. A clustering does not prove that natural psychological types exist: an algorithm can partition a continuous distribution. Its units are cases; factor analysis instead primarily models shared variation among measured variables.

Use logistic regression when the response variable is binary, for example whether a lecturer missed a deadline. It relates the event's log odds to one or more predictors, producing fitted probabilities between zero and one. Exponentiating a coefficient gives an odds ratio for a specified predictor change, conditional on the other model terms. An odds ratio is not generally a ratio of probabilities; the corresponding probability change depends on the starting probability and other predictors.

Report the outcome coding, estimated associations with confidence intervals, and relevant model checks. Sparse events, complete separation and dependent observations can make ordinary estimates or inferential procedures unreliable; repeated observations may need a generalized linear mixed model. For prediction, evaluate calibration and discrimination on data not used to tune the model. An adjusted association in an observational study does not automatically identify a causal effect.

Use factor analysis when several observed measurements may reflect fewer unobserved dimensions, such as related questionnaire items about planning, prioritization and record keeping. A common-factor model separates shared variation represented by latent factors from item-specific variation. Factor loadings describe how measurements relate to those latent factors; communalities summarize their shared-variance contributions.

Exploratory factor analysis investigates a structure not yet fixed; confirmatory factor analysis evaluates prespecified measurement restrictions, preferably in independent data. Extraction, factor count and factor rotation affect interpretation, so report them and justify the sample and measurement assumptions. A statistically convenient factor solution does not alone establish construct validity or demonstrate the existence of a literal psychological mechanism. The distinction from principal component analysis is the common-versus-total-variation objective, and from cluster analysis the modeling of measurement relationships rather than the grouping of cases.

Bootstrapping estimates sampling uncertainty by repeating an analysis on resampled data. In the ordinary nonparametric bootstrap, draw observations with replacement from the empirical sample, retaining its size, and recalculate the statistic many times. The resulting distribution can estimate a standard error, assess bias or form a justified bootstrap confidence interval. It is useful when an analytic sampling distribution is difficult to obtain.

The resampling unit must match the design: keep paired measurements together, and resample independent clusters when observations are clustered. Resampling individual ratings as if they were independent lecturers would misrepresent uncertainty. The ordinary bootstrap assumes the sample and chosen resampling scheme adequately represent the relevant data-generating variation; it does not repair selection bias, guarantee accurate inference from a tiny sample, or succeed for every statistic. Specify the interval method and computational settings, since a simple percentile interval is not universally appropriate.

Preregistration creates a time-stamped record of research questions and planned methods before the relevant results are examined. Specify the design, recruitment and stopping rules, outcome definitions, exclusions, principal tests, model choices and any conditional decision rules. For existing data, disclose prior access and what was already known. A sufficiently precise record helps readers distinguish prior predictions from choices prompted by the results.^[ref]

It does not prohibit useful changes or new discoveries: report departures, their timing and reasons, and distinguish exploratory analysis from confirmatory analysis. A vague or selectively followed registration does not solve selective reporting, and preregistration cannot repair a poor measure or inadequate design of experiments. A registered report adds pre-result peer review and conditional publication acceptance, which are not implied by registration alone.

HARKing means presenting a hypothesis generated after inspecting results as though it had been specified beforehand. For example, after trying several possible factor structures in the lecturer data, a researcher might claim that the best-fitting structure was their original prediction. The problem is the misrepresented evidential history, not the act of discovering a pattern.

Such undisclosed selection makes apparent confirmation too persuasive because the reported test does not represent the whole search that produced the hypothesis. Label new ideas as exploratory, record prior plans through preregistration, disclose the search and evaluate the resulting prediction in independent data. Registered reports and transparent reporting can reduce incentives for that misrepresentation while preserving legitimate exploratory work.^[ref]