Loss Function

What is a loss function?

Loss functions are used to determine the error (aka “the loss”) between the output of our algorithms and the given target value. In layman’s terms, the loss function expresses how far off the mark our computed output is.

Common Loss Functions

There are multiple ways to determine loss. Two of the most popular loss functions in machine learning are the 0-1 loss function and the quadratic loss function. The 0-1 loss function is an indicator function that returns 1 when the target and output are not equal and zero otherwise:

0-1 Loss:

The quadratic loss is a commonly used symmetric loss function. The quadratic losses’ symmetry comes from its output being identical with relation to targets that differ by some value x in any direction (i.e. if the output overshoots by 1, that is the same as undershooting by 1). The quadratic loss is of the following form:

QuadraticLoss: (y,ŷ) = C(y- ŷ)2

In the formula above, C is a constant and the value of C has makes no difference to the decision. C can be ignored if set to 1 or, as is commonly done in machine learning, set to ½ to give the quadratic loss a nice differentiable form.

Applications of Loss Functions

Loss functions are used in optimization problems with the goal of minimizing the loss. Loss functions are used in regression when finding a line of best fit by minimizing the overall loss of all the points with the prediction from the line. Loss functions are used while training perceptrons and neural networks by influencing how their weights are updated. The larger the loss is, the larger the update. By minimizing the loss, the model’s accuracy is maximized. However, the tradeoff between size of update and minimal loss must be evaluated in these machine learning applications.