Large-Scale Classification using Multinomial Regression and ADMM
share
We present a novel method for learning the weights in multinomial logistic regression based on the alternating direction method of multipliers (ADMM). In each iteration, our algorithm decomposes the training into three steps; a linear least-squares problem for the weights, a global variable update involving a separable cross-entropy loss function, and a trivial dual variable update The least-squares problem can be factorized in the off-line phase, and the separability in the global variable update allows for efficient parallelization, leading to faster convergence. We compare our method with stochastic gradient descent for linear classification as well as for transfer learning and show that the proposed ADMM-Softmax leads to improved generalization and convergence.
READ FULL TEXT