Universal Binary Neural Networks Design by Improved Differentiable Neural Architecture Search

Abstract

Binary Neural Networks (BNNs) using 1-bit weights and activations are emerging as a promising approach for mobile devices and edge computing platforms. Concurrently, traditional Neural Architecture Search (NAS) has gained widespread usage in automatically designing network architectures. However, the computation involved in binary NAS is more complex than in NAS due to the substantial information loss incurred by binary modules, and different binary spaces are required for different tasks. To address these challenges, a universal binary neural architecture search (UBNAS) algorithm is proposed. In this paper, the ApproxSign function is used to reduce the gradient error and accelerate the convergence in binary network searching and training. Moreover, UBNAS adopts a novel search space consisting of operations appropriate for the binary methods. To improve the original space operation module, we explore the effect of diverse structures for various modules and ultimately obtain a universal binary network structure. Additionally, the channel sampling ratio is adjusted to balance the advantages of different operations and an early stopping strategy is implemented to significantly reduce the computational burden associated with searching. We perform extensive experiments on CIFAR10, and ImageNet datasets and the results demonstrate the effectiveness of the proposed method.