Communication-Efficient Federated Learning with Heterogeneous Devices

Abstract

The conventional model aggregation-based federated learning (FL) approaches require all local models to have the same architecture and fail to support practical scenarios with heterogeneous local models. Moreover, the frequent model exchange is costly for resource-limited wireless networks since modern deep neural networks usually have over-million parameters. To tackle these challenges, we first propose a novel knowledge-aided FL (KFL) framework, which aggregates light high-level data features, namely knowledge, in the per-round learning process. The KFL allows devices to design their machine learning models independently and reduces the communication overhead in the training process. We then experimentally show that different temporal device scheduling patterns lead to considerably different learning performance. With this insight, we formulate a stochastic optimization problem for joint device scheduling and bandwidth allocation under limited devices' energy budgets and develop an efficient online algorithm to achieve an energy-learning trade-off in the learning process. Experimental results on the CIFAR-10 dataset show that the proposed KFL can reduce over 87% communication overhead while achieving better learning performance than the baselines. In addition, the proposed device scheduling algorithm converges faster than benchmark scheduling schemes.