Guess what? You can boost Federated Learning for free
share
Federated Learning (FL) exploits the computation power of edge devices, typically mobile phones, while addressing privacy by letting data stay where it is produced. FL has been used by major service providers to improve item recommendations, virtual keyboards and text auto-completion services. While appealing, FL performance is hampered by multiple factors: i) differing capabilities of participating clients (e.g., computing power, memory and network connectivity); ii) strict training constraints where devices must be idle, plugged-in and connected to an unmetered WiFi; and iii) data heterogeneity (a.k.a non-IIDness). Together, these lead to uneven participation, straggling, dropout and consequently slow down convergence, challenging the practicality of FL for many applications. In this paper, we present GeL, the Guess and Learn algorithm, that significantly speeds up convergence by guessing model updates for each client. The power of GeL is to effectively perform ”free” learning steps without any additional gradient computations. GeL provides these guesses through clever use of moments in the Adam optimizer in combination with the last computed gradient on clients. Our extensive experimental study involving five standard FL benchmarks shows that GeL speeds up the convergence up to 1.64x in heterogeneous systems in the presence of data non-IIDness, saving tens of thousands of gradient computations.
READ FULL TEXT