Codex @codex  0

A psychometrician who proposed the random-data approach underlying parallel analysis.^[ref]

Treat the forty-person convenience sample as a pilot, then develop and validate the measure in its intended population. Start by defining whose organizational ability the questionnaire measures and who answers it: lecturer self-reports and student ratings are different measurements. Specify the relevant activities and context. Discuss the candidate items with lecturers and appropriate users, seek expert review, and use cognitive interviewing to find ambiguous wording, double-barreled items and irrelevant content. This establishes content validity rather than assuming that correlations alone identify the intended construct.

Recruit a larger, better-targeted sample covering relevant roles, disciplines and institutions. A convenience sample of adults who cannot meaningfully judge lecturers' organization is not made suitable simply by increasing its size. Plan the sample size using plausible factor loadings, communalities, number of dimensions and indicators, response distributions, missingness and required estimation stability, preferably with simulation. Forty observations for sixteen items offer limited grounds for a stable generalizable structure; no universal participants-per-item cutoff makes this analysis automatically impossible or automatically adequate. If students rate the same lecturers, recognize that the ratings are clustered rather than independent lecturer-level observations.

Check missing responses, item distributions, scoring direction and correlations, identifying redundant or poorly functioning items. If the underlying dimensions are not already well specified, use exploratory factor analysis to model shared variation among items. Principal component analysis instead forms combinations explaining total observed variation; it is not the same latent measurement model. Choose a suitable common-factor estimator, such as principal axis factoring, or maximum likelihood estimation when its assumptions are credible. For ordered response categories, consider an ordinal estimator and polychoric correlations, while checking whether category counts support their estimation. Do not treat those correlations as assumption-free.^[ref]

Choose the number of factors using parallel analysis, the scree plot, substantive interpretation and the quality of the resulting model, rather than an eigenvalue-one rule alone. Parallel analysis compares the observed eigenvalues with a reference generated from uncorrelated data of comparable dimensions; its implementation must match the chosen correlation and extraction approach.^[ref] Use oblique factor rotation if organizational dimensions can plausibly correlate. Inspect factor loadings, cross-loadings, communalities, factor correlations and model discrepancies, and retain enough substantively relevant indicators for each dimension. Do not repeatedly delete items solely to improve numerical fit at the expense of coverage.

After developing the structure, test it using confirmatory factor analysis in an independently recruited validation sample, with the proposed structure fixed beforehand. Splitting the existing forty participants would not solve the shortage of information. Evaluate model fit and uncertainty, measurement reliability, test-retest reliability where stability is intended, and evidence for construct validity using related, distinct and relevant behavioral measures. Check measurement invariance before comparing scores across groups. Report alternative plausible models, all item decisions, sampling limitations and the distinction between an initial exploratory instrument and an independently validated measure.

Make the development process inspectable and separate exploratory decisions from independent confirmation. Create a preregistration for the validation stage before examining its outcome-relevant data, specifying recruitment, stopping rules, exclusions, missing-data treatment, item scoring, estimator, factor structure and criteria for evaluating it. The exploratory stage can also have a recorded protocol, including a planned parallel analysis rule; label data-driven item revisions and model changes as exploratory rather than disguising them as prior predictions. Document deviations and their reasons.^[ref]

Keep an immutable copy of the collected data, record cleaning steps, and provide a data dictionary, final item wording, scoring instructions and executable analysis code. Record software and package versions, random seeds for simulations or bootstrap calculations, and the steps needed to regenerate tables from the permitted data. These support computational reproducibility; an independently collected validation study or scientific replication addresses a different question.

Share appropriately de-identified data where consent and re-identification risk permit, with controlled access or a clear restriction statement where they do not. Protect lecturer and rater identities and disclose the pilot's convenience sampling and all item-selection steps. Report inconclusive and unfavorable findings as well as favorable ones. A registered report can obtain peer review of the validation plan before its results are known, reducing outcome-dependent publication incentives.^[ref]

There is evidence for particular structures of social intelligence, but no single established factor structure independent of definition, measurement method and population. “Not known” should therefore be interpreted as unresolved construct organization, not an absence of factor analyses or proof that social understanding cannot be measured. A factor is a proposed source of shared variation among selected measurements; it does not acquire a psychological meaning simply because a statistical extraction produces it.

The first difficulty is defining what belongs in the construct. Understanding another person's intentions, remembering social information, knowing social conventions and behaving effectively in an interaction are distinguishable achievements. They may draw on partly different abilities. A questionnaire asking whether one feels socially competent also measures self-perception and typical behavior, whereas a scored performance task asks what one can do under specified conditions. Related constructs such as general intelligence, personality and theory of mind overlap with some definitions but are not interchangeable with every proposed social ability. Different definitions therefore generate different item domains before factor analysis even begins.

Historical negative findings demonstrate the problem of discriminant evidence. Daniel P. Keating studied 117 college students using three academic-intelligence measures and three proposed social-intelligence measures. Correlations within the social set were not higher than correlations across the social and academic sets, and factor analysis did not identify a separate social factor. This challenged the coherence and distinctiveness of that test battery. It did not logically refute every possible social-intelligence construct: the chosen measures, restricted student sample and criteria determine what the negative result can exclude.^[ref]

Conversely, a positive structure within one instrument does not settle the whole domain. David H. Silvera and colleagues developed the Tromsø Social Intelligence Scale through studies of expert interpretations, item selection and stability, obtaining a 21-item self-report instrument with three dimensions: social information processing, social skills and social awareness. This provides a usable operationalization and evidence for internal organization. But the structure of self-descriptions is not automatically the structure of maximal social performance. Shared response style, self-esteem, item wording and perceived competence may generate covariation that a performance test would not reproduce.^[ref]

A different strategy separates content from method. Susanne Weis and colleagues tested a performance model distinguishing social understanding, memory and knowledge using verbal, pictorial and video-based tasks in a multitrait-multimethod design. In 118 students, confirmatory analyses supported multidimensional organization and provided evidence of differentiation from academic intelligence, although social-memory measures showed overlap. This is positive evidence against the claim that no structure has ever been found. The factor labels differ from the self-report scale because the proposed processes and tasks differ; the fact that both studies obtained three dimensions does not make the dimensions equivalent.^[ref]

These studies expose the need to distinguish construct validity from attractive factor labels. A social story task may mainly depend on reading comprehension; a facial or video judgment may introduce perceptual and memory demands. If all putative measures of one trait use the same format, a “trait factor” can actually be a method factor. Multiple methods for each proposed trait allow stronger tests of convergent validity and discriminant validity, although even an MTMM model depends on identifying restrictions and adequate indicators. A good model should explain cross-method covariance as well as within-format similarity.

Measurement quality also changes the observed structure. Unreliable tasks attenuate correlations, potentially hiding a shared ability; overlapping items or shared response biases can inflate them. Social performance depends on cultural conventions and context, and a scoring key based on majority agreement need not establish objectively superior understanding in every setting. Small or homogeneous samples can produce unstable loadings or restrict ability variance. Evidence for measurement invariance across groups and contexts is therefore needed before interpreting different scores or factor solutions as differences in the same underlying construct.

Statistical choices contribute additional uncertainty. Exploratory factor analysis models common variation, whereas principal component analysis summarizes total variance; their dimensions answer different questions. Factor retention, residual correlations and oblique factor rotation affect interpretation. In a general factor model,where $\Phi$ is factor covariance and $\Psi$ residual covariance. For invertible $T$, replacing $\Lambda$ with $\Lambda T$ and $\Phi$ with $T^{-1}\Phi T^{-T}$ leaves $\Sigma$ unchanged. Hence covariance fit alone cannot uniquely name unrestricted factors. Identifying restrictions and substantive predictions are needed. This rotational issue concerns orientation; it does not by itself imply that every possible number of factors fits equally well.

A stronger research program would specify the proposed abilities before choosing indicators, obtain multiple reliable measures across methods and social contexts, and measure competing general-intelligence and personality constructs. Exploratory work and parallel analysis can guide model development; a prespecified confirmatory factor analysis in an independent, adequately varied sample provides a separate test. Competing correlated-factor, hierarchical and method-factor models should be compared, with appropriate attention to identification and out-of-sample performance. Predicting consequential social outcomes beyond competing constructs adds evidence that internal consistency alone cannot supply.

The unresolved structure reflects construct and measurement choices as well as statistical uncertainty. Existing studies support specific, testable organizations, but universal agreement requires convergence across performance and report methods, populations and external criteria. The productive conclusion is to improve competing models and their tests, rather than treating one questionnaire's factors as a final inventory of social intelligence or interpreting an early null result as proof of impossibility.