Asynchronous Distributed Optimization with Randomized Delays
share
In this work, we study asynchronous finite sum minimization in a distributed-data setting with a central parameter server. While asynchrony is well understood in parallel settings where the data is accessible by all machines, little is known for the distributed-data setting. We introduce a variant of SAGA called ADSAGA for the distributed-data setting where each machine stores a partition of the data. We show that with independent exponential work times – a common assumption in distributed optimization – ADSAGA converges in Õ((n + √(m)κ)log(1/ϵ)) iterations, where n is the number of component functions, m is the number of machines, and κ is a condition number. We empirically compare the iteration complexity of ADSAGA to existing parallel and distributed algorithms, including synchronous mini-batch algorithms.
READ FULL TEXT