Communication-Efficient Federated Learning through Importance Sampling
share
The high communication cost of sending model updates from the clients to the server is a significant bottleneck for scalable federated learning (FL). Among existing approaches, state-of-the-art bitrate-accuracy tradeoffs have been achieved using stochastic compression methods – in which the client n sends a sample from a client-only probability distribution q_ϕ^(n), and the server estimates the mean of the clients' distributions using these samples. However, such methods do not take full advantage of the FL setup where the server, throughout the training process, has side information in the form of a pre-data distribution p_θ that is close to the client's distribution q_ϕ^(n) in Kullback-Leibler (KL) divergence. In this work, we exploit this closeness between the clients' distributions q_ϕ^(n)'s and the side information p_θ at the server, and propose a framework that requires approximately D_KL(q_ϕ^(n)|| p_θ) bits of communication. We show that our method can be integrated into many existing stochastic compression frameworks such as FedPM, Federated SGLD, and QSGD to attain the same (and often higher) test accuracy with up to 50 times reduction in the bitrate.
READ FULL TEXT