Higher-Order Corrections to Optimisers based on Newton's Method
share
The Newton, Gauss–Newton and Levenberg–Marquardt methods all use the first derivative of a vector function (the Jacobian) to minimise its sum of squares. When the Jacobian matrix is ill-conditioned, the function varies much faster in some directions than others and the space of possible improvement in sum of squares becomes a long narrow ellipsoid in the linear model. This means that even a small amount of nonlinearity in the problem parameters can cause a proposed point far down the long axis of the ellipsoid to fall outside of the actual curved valley of improved values, even though it is quite nearby. This paper presents a differential equation that `follows' these valleys, based on the technique of geodesic acceleration, which itself provides a 2^nd order improvement to the Levenberg–Marquardt iteration step. Higher derivatives of this equation are computed that allow n^th order improvements to the optimisation methods to be derived. These higher-order accelerated methods up to 4^th order are tested numerically and shown to provide substantial reduction of both number of steps and computation time.
READ FULL TEXT