 FocusedILS

FocusedILS is the ParamILS variant that adapts the number of evaluation runs assigned to different configurations. Its pairwise comparisons use common benchmark instances and random seeds, extending the evidence when necessary instead of comparing arbitrary means from different run lists.^[ref]

An original example shows why unequal evidence needs care. Suppose configuration $A$ has completed two runs with costs $(2,2)$ seconds, while $B$ has four costs $(3,3,30,30)$ seconds. Their available means, $2$ and $16.5$, mix different instance coverage. On the two shared runs the comparison is $2$ against $3$. If extending $A$ to the same four runs gives $(2,2,40,40)$, its mean becomes $21$, reversing the ranking against $B$'s $16.5$. Matching the evidence does not prevent a ranking from changing when more evidence arrives.

The run allocation and the configuration search are different layers. Iterated local search proposes which settings to compare; the FocusedILS comparison procedure determines how much measured evidence supports the decision. BasicILS instead uses a fixed evaluation count $N$. Both still depend on a declared cost statistic and a representative workload.

The example is a finite-data illustration, not a confidence interval or proof about unseen inputs. A setting selected after adaptive evaluation should be assessed on unused instances, with the full tuning budget recorded. The cost of finding the configuration and the subsequent cost of running it answer different questions.