Measurement reliability 2026-10-24
Consistency or precision of measurements under stated sources of variation. Reliability depends on the population, conditions and intended score; it is not a permanent property of an instrument alone. Inter-rater reliability and test-retest reliability address different sources of variation. Reliability does not establish construct validity.
Improve the sampling, measurement and dependence model before interpreting the association. Define the target population and recruit a broader sample with a transparent sampling frame, ideally using probability sampling where feasible and adequate representation across the relevant ages and roles. Justify the sample size for the intended effect size and precision. Preselection on reputation or the outcomes should be avoided or explicitly modeled.
Operationalize workplace friendliness with several relevant measures, such as independently obtained colleague reports and standardized behavioral tasks, rather than relying only on photographs. Use standardized photographs if impressions are themselves the research target, several independent raters, and blinding to power and the specific hypothesis. Measure inter-rater reliability and retain the distinction between impression and behavior. Establish what the power measure actually assesses and record plausible confounders with a causal rationale.
For a linear regression analysis, use friendliness as an explicitly declared outcome and power as a continuous predictor. Retain actual age as a continuous covariate when available instead of treating three age groups as equally spaced measurements; consider justified nonlinearity. If only age categories are available, use explicit categorical contrasts and nonoverlapping category definitions. Check regression diagnostics, influential observations, heteroscedasticity and multicollinearity. Account for executive and rater dependence with an appropriate mixed-effects model when repeated or crossed ratings are analyzed; do not count each rating as a new executive. Report estimates with confidence intervals and disclose exclusions and missing observations.
Preregistration can distinguish planned analyses from exploratory ones. A longitudinal study with repeated measures can establish temporal ordering more clearly, though it remains vulnerable to confounding. A feasible, ethical randomized experiment manipulating assigned power or role in a controlled task could test a more narrowly defined causal effect; its external validity to senior executives must still be assessed. No observational adjustment alone guarantees removal of all unmeasured alternatives.