LSOS: Line-search Second-Order Stochastic optimization methods

07/31/2020
by   Daniela di Serafino, et al.
0

We develop a line-search second-order algorithmic framework for optimization problems in noisy environments, i.e., assuming that only noisy values are available for the objective function and its gradient and Hessian. In the general noisy case, almost sure convergence of the methods fitting into the framework is proved when line searches and suitably decaying step lengths are combined. When the objective function is a finite sum, such as in machine learning applications, our framework is specialized as a stochastic L-BFGS method with line search only, with almost sure convergence to the solution. In this case, linear convergence rate of the expected function error is also proved, along with a worst-case 𝒪 (log(ε^-1)) complexity bound. Numerical experiments, including comparisons with state-of-the art first- and second-order stochastic optimization methods, show the efficiency of our approach.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset