Gaussian-discrete restricted Boltzmann machine with sparse-regularized hidden layer

Abstract

Abstract Overfitting is a critical concern in machine learning, particularly when the representation capabilities of learning models surpass the complexities present in the training datasets. To mitigate overfitting, curtailing the representation power of the model through suitable techniques such as regularization is necessary. In this study, a sparse-regularization method for Gaussian–Discrete restricted Boltzmann machines (GDRBMs) is considered. A GDRBM is a variant of restricted Boltzmann machines that comprises a continuous visible layer and discrete hidden layer. In the proposed model, sparse GDRBM (S-GDRBM), a sparse prior that encourages sparse representations of the hidden layer is employed. The strength of the prior (i.e., the sparse-regularization strength) can be tuned within the standard scenario of maximum likelihood learning; that is, the strength can be adaptively tuned based on the complexities of the datasets during training. We validated the proposed S-GDRBM using numerical experiments.