Ciro Santilli @cirosantilli 37

A parameter that you choose which determines how the algorithm will perform.

In the case of machine learning in particular, it is not part of the training data set.

Hyperparameters can also be considered in domains outside of machine learning however, e.g. the step size in partial differential equation solver is entirely independent from the problem itself and could be considered a hyperparamter. One difference from machine learning however is that step size hyperparameters in numerical analysis are clearly better if smaller at a higher computational cost. In machine learning however, there is often an optimum somewhere, beyond which overfitting becomes excessive.

The extreme overfitting case of training is to have a map where each input leads to one output.

However it is cool that this overfit does not allow you to compute the final input for which there is no known output.

This therefore forces the creation of more general solution rules.

While in some cases solutions can work for any input, in many others they require specific assumptions about input, but the model could simply check that the assumptions apply to all inputs and use them for the final algorithm.

Bibliography:

Philosophically, machine learning can be seen as a form of lossy compression.

And if we make it too lossless, then we are basically overfitting.

Bibliography: