Gradient Descent-Type Methods: Background and Simple Unified Convergence Analysis

12/19/2022
by   Quoc Tran-Dinh, et al.
0

In this book chapter, we briefly describe the main components that constitute the gradient descent method and its accelerated and stochastic variants. We aim at explaining these components from a mathematical point of view, including theoretical and practical aspects, but at an elementary level. We will focus on basic variants of the gradient descent method and then extend our view to recent variants, especially variance-reduced stochastic gradient schemes (SGD). Our approach relies on revealing the structures presented inside the problem and the assumptions imposed on the objective function. Our convergence analysis unifies several known results and relies on a general, but elementary recursive expression. We have illustrated this analysis on several common schemes.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset