= Solution
\b[Treat the forty-person convenience sample as a pilot, then develop and validate the measure in its intended population.] Start by defining whose organizational ability the <questionnaire> measures and who answers it: lecturer self-reports and student ratings are different measurements. Specify the relevant activities and context. Discuss the candidate items with lecturers and appropriate users, seek expert review, and use <cognitive interviewing> to find ambiguous wording, double-barreled items and irrelevant content. This establishes <content validity> rather than assuming that correlations alone identify the intended construct.
Recruit a larger, better-targeted sample covering relevant roles, disciplines and institutions. A <convenience sample> of adults who cannot meaningfully judge lecturers' organization is not made suitable simply by increasing its size. Plan the <sample size> using plausible <factor loadings>, <communalities>, number of dimensions and indicators, response distributions, missingness and required estimation stability, preferably with simulation. Forty observations for sixteen items offer limited grounds for a stable generalizable structure; \b[no universal participants-per-item cutoff makes this analysis automatically impossible or automatically adequate]. If students rate the same lecturers, recognize that the ratings are clustered rather than independent lecturer-level observations.
Check missing responses, item distributions, scoring direction and <correlations>, identifying redundant or poorly functioning items. If the underlying dimensions are not already well specified, use <exploratory factor analysis> to model shared variation among items. <Principal component analysis> instead forms combinations explaining total observed variation; it is not the same latent measurement model. Choose a suitable common-factor estimator, such as <principal axis factoring>, or <maximum likelihood estimation> when its assumptions are credible. For ordered response categories, consider an ordinal estimator and <polychoric correlations>, while checking whether category counts support their estimation. Do not treat those correlations as assumption-free.<Principal Components and Exploratory Factor Analysis with SPSS by University of California, Los Angeles>[Common Factor Analysis; comparison with principal components; extraction methods]{ref}
Choose the number of factors using <parallel analysis>, the <scree plot>, substantive interpretation and the quality of the resulting model, rather than an eigenvalue-one rule alone. <Parallel analysis> compares the observed <eigenvalues> with a reference generated from uncorrelated data of comparable dimensions; its implementation must match the chosen correlation and extraction approach.<A rationale and test for the number of factors in factor analysis by John L. Horn>[1965, abstract: random-variable procedure to assess sampling contributions to latent roots]{ref} Use <oblique factor rotation> if organizational dimensions can plausibly correlate. Inspect <factor loadings>, cross-loadings, <communalities>, factor correlations and model discrepancies, and retain enough substantively relevant indicators for each dimension. Do not repeatedly delete items solely to improve numerical fit at the expense of coverage.
After developing the structure, test it using <confirmatory factor analysis> in an independently recruited validation sample, with the proposed structure fixed beforehand. Splitting the existing forty participants would not solve the shortage of information. Evaluate model fit and uncertainty, <measurement reliability>, <test-retest reliability> where stability is intended, and evidence for <construct validity> using related, distinct and relevant behavioral measures. Check <measurement invariance> before comparing scores across groups. Report alternative plausible models, all item decisions, sampling limitations and the distinction between an initial exploratory instrument and an independently validated measure.
Back to article page
