 Model recovery

Model recovery tests whether a model selection procedure can identify the known model that generated simulated data. Generate observations from each candidate statistical model, fit the candidates using the intended pipeline, and record which candidate is selected. It is a check on the comparison procedure under stated simulation conditions.^[ref]

This differs from parameter recovery, which compares estimated parameters with the settings used to generate data within a model. Correctly identifying a model does not automatically imply accurate parameter estimation, and accurate fitting within one model does not establish that a procedure can distinguish it from its competitors.

For an invented example, suppose one hundred simulated datasets come from each of models A and B. The resulting counts areThis confusion matrix estimates selection behavior conditional on the generating model. Its empirical rate of choosing B when A generated is $20/100=0.20$.

The reverse question has a different denominator. Among the $85$ datasets for which B was selected, $65$ were generated by B, giving $65/85\simeq 0.765$. That calculation uses the example's balanced simulation design. It is not interchangeable with the $0.65$ recovery rate for B-generated datasets. Different generating-model frequencies can change the reverse proportion even if the conditional selection behavior stays fixed.^[ref]

Recovery depends on parameter ranges, observation count, noise and the selection criterion. Models with nearly indistinguishable predictions can be hard to recover despite successful numerical optimization. Repeat simulations in the regimes relevant to the planned inference. Finally, recovery among a chosen candidate set does not prove that the real system belongs to that set; checking how well the selected model reproduces relevant observations remains a separate task.^[ref]