Past exam of the experimental design and methods course of the year 2 of the psychology (experimental) course of the University of Oxford / 2022 / Section A

TODO: the supplementary sheet is needed to identify the analysis actually conducted. A two-factor factorial design is described, but that alone does not distinguish an independent-groups two-way analysis of variance, a repeated-measures analysis of variance, a mixed design, or a mixed-effects model. The model call, error structure, participant allocation and output headings would establish which analysis was used. In particular, the prose does not establish that every participant experienced every condition.

The two independent variables are social-cue congruence, with congruent and incongruent levels, and cue-to-response delay, with short ($250$ ms) and long ($750$ ms) levels. The response variable is reproduction reaction time, measured in seconds. Thus the known treatment structure is a $2\times 2$ factorial design, potentially estimating two main effects and their interaction.

TODO: identify the allocation from the missing supplementary sheet or protocol. The eighty recruited participants do not establish whether either factor is within-subjects or between-subjects, whether all observations were retained, or how trials were summarized. The design cannot responsibly be called fully repeated-measures, fully independent-groups or mixed without that information.

TODO: the effect-specific $F$ values are unavailable without the supplementary output. Retrieve the rows for congruence, delay and congruence-by-delay interaction, if those terms were fitted. In an appropriate analysis of variance, each F-test compares a term's mean square in ANOVA with its designated error mean square in ANOVA; repeated observations can require different error terms. Neither the factor labels nor the recruited sample size determines those ratios.

TODO: neither the significant effects nor their numerical effect sizes can be determined. Inspect the output's significance results and the definition of its reported effect size. If it supplies partial eta squared, that quantity is $S{S}_{\text{effect}}/(S{S}_{\text{effect}}+S{S}_{\text{error for effect}})$; it is not automatically ordinary eta squared or a fraction of all observed variation. Repeated-measures conventions and the selected error term matter. Report the actual metric and preferably an uncertainty interval; do not manufacture an effect size from the number of participants.

TODO: retrieve both numerator and denominator statistical degrees of freedom for each significant effect from the missing output. The table is also needed to establish which effects, if any, meet the analysis's significance criterion. Participant allocation, exclusions, aggregation and fitted error structure determine the applicable statistical degrees of freedom; the recruitment count is not enough to reconstruct them. No numerical statistical degrees of freedom are asserted here.

TODO: an empirical Results section cannot be completed without the analysis and descriptive output. It must identify the actual factorial design, outcome scale and fitted model; summarize the four condition means with appropriately labeled standard deviations or standard errors; and report the congruence, delay and interaction tests with their actual F-test statistics, statistical degrees of freedom, p-values, effect sizes and relevant confidence intervals.

Interpret an interaction as evidence that the cue-congruence difference varies with delay, not merely as evidence that one separate test was significant and another was not. Any simple-effects analysis must identify its contrasts and multiple testing treatment. If the analysis used transformed reaction times, distinguish the model's scale from raw-second summaries. A nonsignificant result does not establish equality or absence of a theoretically meaningful effect. These are reporting requirements, not placeholders for invented results.

TODO: whether additional analyses are needed depends on the unavailable results and prespecified questions. An estimated congruence-by-delay interaction can be clarified by the cue-congruence contrast at each delay, with confidence intervals and justified multiple testing control. A direct interaction test addresses whether those contrasts differ; comparing their separate significance labels does not.

Even without a significant interaction, an adequately powered, prespecified contrast may address a distinct scientific question; indiscriminate testing until something is significant is not justified. Inspect regression diagnostics and the treatment of dependent trials where appropriate. Without the output, one cannot decide that any particular follow-up is required or conclude that none is necessary.

TODO: the missing condition means and inferential output prevent a conclusion about whether congruent gaze changed reaction time or whether that change depended on delay. The direction, magnitude and uncertainty of those two comparisons are needed before drawing conclusions about attentional guidance in this task.

TODO: the missing output is needed to name the test actually used. A correlation, multiple linear regression adjusting for age, or another model would answer different questions. The presence of power, friendliness and age variables does not identify which were included, which was the response variable, how age was encoded, or whether an interaction was tested. Read the model call and coefficient or test table before identifying the procedure.

TODO: the numerical results, association direction and statistical evidence are unavailable. The required output includes the actual fitted model, estimates and uncertainty, test statistics, statistical degrees of freedom, p-values and an appropriate model-fit summary. An age-adjusted regression coefficient, if reported, would describe a conditional association under that model rather than the same estimand as an unadjusted correlation. Irrespective of its numerical outcome, this observational study does not by itself show that executive power changes real-life friendliness.

Several mechanisms could explain an observed association without a causal effect of power on friendliness. No exposure was assigned, so this is an observational study. Age, organizational role, wealth, gender, industry, public prominence or other common causes could produce confounding. Reverse direction is also possible: interpersonal behavior might influence advancement and hence power. Adjusting for age alone would not remove every alternative explanation.

The measured outcome is perceived friendliness from a photograph. Facial expression, pose, lighting, clothing, attractiveness and familiarity could affect those ratings independently of everyday behavior. If raters recognize an executive or know their status, expectations could alter the outcome; financial experts' power ratings may themselves involve judgment and measurement error. These are construct validity issues, since a photographic impression need not measure friendliness toward colleagues.

The preselected thirty executives are not a probability sample of all workers or even necessarily of all executives. Selection bias, narrow representation and restriction of range could alter associations and limit external validity. Multiple ratings of the same executive are also not independent executive observations; ignoring that structure would be pseudoreplication. These possibilities apply whether the reported association is positive, negative or inconclusive; the unavailable numerical findings are not assumed.

Improve the sampling, measurement and dependence model before interpreting the association. Define the target population and recruit a broader sample with a transparent sampling frame, ideally using probability sampling where feasible and adequate representation across the relevant ages and roles. Justify the sample size for the intended effect size and precision. Preselection on reputation or the outcomes should be avoided or explicitly modeled.

Operationalize workplace friendliness with several relevant measures, such as independently obtained colleague reports and standardized behavioral tasks, rather than relying only on photographs. Use standardized photographs if impressions are themselves the research target, several independent raters, and blinding to power and the specific hypothesis. Measure inter-rater reliability and retain the distinction between impression and behavior. Establish what the power measure actually assesses and record plausible confounders with a causal rationale.

For a linear regression analysis, use friendliness as an explicitly declared outcome and power as a continuous predictor. Retain actual age as a continuous covariate when available instead of treating three age groups as equally spaced measurements; consider justified nonlinearity. If only age categories are available, use explicit categorical contrasts and nonoverlapping category definitions. Check regression diagnostics, influential observations, heteroscedasticity and multicollinearity. Account for executive and rater dependence with an appropriate mixed-effects model when repeated or crossed ratings are analyzed; do not count each rating as a new executive. Report estimates with confidence intervals and disclose exclusions and missing observations.

Preregistration can distinguish planned analyses from exploratory ones. A longitudinal study with repeated measures can establish temporal ordering more clearly, though it remains vulnerable to confounding. A feasible, ethical randomized experiment manipulating assigned power or role in a controlled task could test a more narrowly defined causal effect; its external validity to senior executives must still be assessed. No observational adjustment alone guarantees removal of all unmeasured alternatives.

Use one-way analysis of variance to compare a quantitative response variable across levels of one categorical factor; for example, compare task accuracy across three independently assigned teaching methods. The ordinary independent-groups procedure tests the null hypothesis that all group population means are equal. Its F-test compares variation associated with the group factor against within-group error variation; a significant result indicates that not all population means are equal, without identifying which groups differ.^[ref]

The classical one-way normal linear model assumes independent observations and within-group errors approximately following a normal distribution with a common variance. Repeated observations on the same people require a different dependence structure; pronounced unequal variances may call for Welch's one-way analysis of variance. Planned contrasts or appropriately controlled follow-up comparisons identify scientifically relevant differences. Report the means, differences and confidence intervals, not just significance. Causal interpretation depends on the design of experiments, particularly randomization, rather than on the F-test alone.

Use a bivariate correlation to describe the direction and strength of association between two measured variables, such as study time and test score. The Pearson correlation coefficient describes linear association between two quantitative variables: its sign gives direction, and its magnitude reflects linear association on a standardized scale. A confidence interval describes estimation uncertainty; a corresponding hypothesis test can assess a specified null association under its sampling assumptions.

Inspect a scatter plot for curvature, clusters, influential observations and restriction of range. A coefficient near zero does not exclude a strong nonlinear relationship. For ordered or markedly nonnormal measurements, a Spearman rank correlation can describe monotone association, using ranks; it does not require a linear relation in the original units. Ordinary inferential procedures require independent observational units and appropriate assumptions for their chosen method. Correlation alone does not identify a causal direction or exclude confounding.

Use a Student t-test to assess a mean or mean difference relative to its standard error. A one-sample version compares a population mean with a specified reference. An independent two-sample version compares two separate groups, such as groups assigned different instructions; Welch's t-test permits unequal group variances. A paired t-test uses within-pair differences, such as each participant's before/after scores, rather than pretending those measurements are independent.^[ref]

The relevant normality assumption concerns within-group errors for the independent-groups model or the distribution of within-pair differences for the paired model. Independent participants or pairs remain essential. Report the estimated difference and its confidence interval together with the test result. A small p-value indicates incompatibility with the specified null hypothesis under the model; it does not measure the size of the difference, the probability that the null hypothesis is true, or whether the difference matters practically.

Qualitative research methods investigate meanings, experiences and processes in context. For example, interviews about why lecturers find scheduling difficult could reveal unanticipated constraints that a fixed-response questionnaire omits. Interviews, focus groups, observations and documents provide material; a clearly stated approach such as thematic analysis or qualitative content analysis organizes its interpretation. These methods can develop an explanation or refine a construct rather than estimate a population prevalence.

Recruit participants whose experiences address the research question, for example by purposive sampling across different teaching roles, and explain the basis for the sample's adequacy. Record and transcribe material where appropriate and agreed. Describe how the analysis moved from material to codes, interpretations and claims; support interpretations with contextualized, confidentiality-preserving extracts. Consider contrary cases, alternative interpretations and the researcher's role through researcher reflexivity. The criteria for quality should fit the analytic approach: agreement between coders is useful in some structured coding studies, but is not a universal validity criterion for every interpretive method. Informed consent and protection against identification are especially important when detailed accounts concern identifiable colleagues.

The normality assumption concerns the model's error distribution, not necessarily the pooled raw observations. In an independent-groups analysis of variance, errors around each group mean are modeled as following a normal distribution; combining groups with genuinely different means can produce a nonnormal pooled distribution even when this assumption holds. In a paired t-test, the relevant distribution is that of within-pair differences. Linear regression conventionally assumes normal errors conditional on the predictors for exact small-sample Student t-tests and F-tests.^[ref]

Inspect suitable within-group or fitted-model regression residuals using a quantile-quantile plot and examine outliers and asymmetry. A formal normality test is not a decisive automatic gatekeeper: with small samples it may miss important departures, while large samples can detect minor departures. Approximate robustness depends on the sample size, balance, tail behavior and intended inference. The central limit theorem concerns sampling distributions and does not make every raw observation normal or repair dependence. Depending on the question, a justified transformation, a better-fitting probability model, robust inference or an appropriate bootstrap may be preferable. The normality assumption, independence and equal variance are distinct assumptions.

Research ethics governs how a worthwhile question is investigated while respecting those involved. The British Psychological Society's framework emphasizes respect for autonomy, privacy and dignity, scientific integrity, social responsibility, and maximizing benefit while minimizing harm. Seek appropriate independent ethical review before recruitment and assess risks in the actual study context.^[ref]

Give understandable information, seek voluntary informed consent, explain the practical limits of withdrawal, and avoid pressure arising from a teacher–student or employer–employee relationship. Limit identifiable information, control access and distinguish confidential storage from genuine anonymity. Any justified withholding or research deception needs proportionate review and appropriate research debriefing; debriefing does not undo every possible harm. Explain relevant procedures and offer questions and support consistent with the study's risks.^[ref]

In the lecturer questionnaire study, make participation independent of employment evaluation and prevent identifiable responses reaching supervisors. Ethical reporting also requires honest analysis, disclosure of limitations and proportionate data sharing: open science does not require exposing confidential participant records.