Deep Neural Network Optimization Based on Binary Method for Handling Multi-Class Problems

Abstract

In this paper, we conceive a new kind of output layer design in deep neural networks for the multi-class problems. The traditional output layer is set by the one-to-one method. For the one-to-one method, the output layer neuron number is the same as the class number. And the ideal output for the j-th class sample is $e_{j}$ , where $e_{j}$ is j-th unit vector. However, one-to-one method requires too many output neurons, which will increase the number of weights connecting the last-hidden and the output layers. Furthermore, during the process of network training, computation time and cost will greatly increase. We design the binary method for the output layer: Let the class number be k ( $k\geq 3$ ), and $2^{a-1} < k \le 2^{a} \,\,({a=\lceil log_{2}k \rceil })$ , then the output layer neuron number is a and the ideal output is designed by binary method. Obviously, the binary method uses less output nodes than the traditional one-to-one method. On this foundation, the number of hidden-output weights will also decrease. On the other hand, while training the deep neural network, the learning efficiency will also be significantly improved. Numerical experiments show that binary method has better classification performance and calculation speed than one-to-one method on the datasets.