Stability of Accuracy for the Training of DNNs Via the Uniform Doubling Condition
share
We study the stability of accuracy for the training of deep neural networks. Here the training of a DNN is preformed via the minimization of a cross-entropy loss function and the performance metric is the accuracy (the proportion of objects classified correctly). While training amounts to the decrease of loss, the accuracy does not necessarily increase during the training. A recent result by Berlyand, Jabin and Safsten introduces a doubling condition on the training data which ensures the stability of accuracy during training for DNNs with the absolute value activation function. For training data in ^n, this doubling condition is formulated using slabs in ^n and it depends on the choice of the slabs. The goal of this paper is twofold. First to make the doubling condition uniform, that is independent on the choice of slabs leading to sufficient conditions for stability in terms of training data only. Second to extend the original stability results for the absolute value activation function to a broader class of piecewise linear activation function with finitely many critical points such as the popular Leaky ReLU.
READ FULL TEXT