Communication-Efficient Robust Federated Learning with Noisy Labels

Abstract

Federated learning (FL) is a promising privacy-preserving machine learning\nparadigm over distributed located data. In FL, the data is kept locally by each\nuser. This protects the user privacy, but also makes the server difficult to\nverify data quality, especially if the data are correctly labeled. Training\nwith corrupted labels is harmful to the federated learning task; however,\nlittle attention has been paid to FL in the case of label noise. In this paper,\nwe focus on this problem and propose a learning-based reweighting approach to\nmitigate the effect of noisy labels in FL. More precisely, we tuned a weight\nfor each training sample such that the learned model has optimal generalization\nperformance over a validation set. More formally, the process can be formulated\nas a Federated Bilevel Optimization problem. Bilevel optimization problem is a\ntype of optimization problem with two levels of entangled problems. The\nnon-distributed bilevel problems have witnessed notable progress recently with\nnew efficient algorithms. However, solving bilevel optimization problems under\nthe Federated Learning setting is under-investigated. We identify that the high\ncommunication cost in hypergradient evaluation is the major bottleneck. So we\npropose \\textit{Comm-FedBiO} to solve the general Federated Bilevel\nOptimization problems; more specifically, we propose two\ncommunication-efficient subroutines to estimate the hypergradient. Convergence\nanalysis of the proposed algorithms is also provided. Finally, we apply the\nproposed algorithms to solve the noisy label problem. Our approach has shown\nsuperior performance on several real-world datasets compared to various\nbaselines.\n