A mortized inference for structured deep generative models

Abstract

The development of Artificial Intelligence aims at building systems that can learn to accomplish tasks that human beings or animals can perform. Recent advances in deep learning have demonstrated that neural networks can achieve remarkable progress when employing large amounts of data and computation. In the past ten years, we have seen the rise of various deep generative models that are particularly successful in generation tasks in the domains of computer vision and natural language processing. On the other hand, many inference tasks would involve representing specific structures from data. In the absence of any supervision, we have increasingly seen limitations of standard deep generative models when tackling such inference tasks. In addition to interpreting the structure from data, we also hope to build models that can generalize to sparse data, prevent over-confident predictions, and perform inference in a computation-efficient manner. This thesis explores one possible direction for solving problems where we need more task-specific models by combining probabilistic modeling with deep learning techniques. We achieve that by inducing inductive biases in the form of structured priors in deep generative models. Our hope is that the models incorporate useful domain knowledge that helps to improve generalization and characterize uncertainty. We start by discussing how we can disentangle different factors of variation using Variational Autoencoders (VAEs), where we develop a novel decomposition of the objective to achieve this. Then we will focus on the more structured problems where we will talk about incorporating domain knowledge by inductively biasing the deep generative models with structured priors. We propose Amortized Population Gibbs Samplers, a class of iterative amortized inference methods that can scale to structured models while standard inference methods fail. Then we will generalize this approach and propose Nested Variational Inference, a class of inference methods that combine nested importance sampling with variational inference. Also, we build Inference Combinators where we propose constructors that allow us to more easily implement advanced amortized importance samplers. We also explore a relatively orthogonal topic where we will discuss what good representations mean. We develop Conjugate Energy-Based Models, an alternative to VAEs for unsupervised representation learning. This class of models can learn representations that can align better with class labels in image data compared to baselines.--Author's abstract