= Model recovery
<Model recovery> tests whether a <model selection> procedure can identify the known model that generated simulated data. Generate observations from each candidate <statistical model>, fit the candidates using the intended pipeline, and record which candidate is selected. It is a check on the comparison procedure under stated simulation conditions.<Ten simple rules for the computational modeling of behavioral data by Robert C. Wilson and Anne G. E. Collins>[Can you arbitrate between different models?, model-recovery discussion following Eq. (10)]{ref}
This differs from <parameter recovery>, which compares estimated <parameters> with the settings used to generate data within a model. Correctly identifying a model does not automatically imply accurate <parameter estimation>, and accurate fitting within one model does not establish that a procedure can distinguish it from its competitors.
For an invented example, suppose one hundred simulated datasets come from each of models A and B. The resulting counts are
$$
\begin{array}{c|rr}
&\text{select A}&\text{select B}\\\hline
\text{generate A}&80&20\\
\text{generate B}&35&65
\end{array}.
$$
This <confusion matrix> estimates selection behavior conditional on the generating model. Its empirical rate of choosing B when A generated is $20/100=0.20$.
The reverse question has a different denominator. Among the $85$ datasets for which B was selected, $65$ were generated by B, giving $65/85\simeq0.765$. That calculation uses the example's balanced simulation design. It is not interchangeable with the $0.65$ recovery rate for B-generated datasets. Different generating-model frequencies can change the reverse proportion even if the conditional selection behavior stays fixed.<Ten simple rules for the computational modeling of behavioral data by Robert C. Wilson and Anne G. E. Collins>[A note on interpreting the confusion matrix; Appendix 3, reverse inference]{ref}
Recovery depends on parameter ranges, observation count, noise and the selection criterion. Models with nearly indistinguishable predictions can be hard to recover despite successful numerical optimization. Repeat simulations in the regimes relevant to the planned inference. Finally, recovery among a chosen candidate set does not prove that the real system belongs to that set; checking how well the selected model reproduces relevant observations remains a separate task.<Ten simple rules for the computational modeling of behavioral data by Robert C. Wilson and Anne G. E. Collins>[Be careful with the choice of parameters; The elephant in the room with model comparison]{ref}
Back to article page
