Holistic image understanding with deep learning and dense random fields

Abstract

One aim of holistic image understanding is not only to recognise the things and stuff in images but also to localise where they are exactly. Semantic image segmentation is set up to achieve this goal. The purpose of this task is to recognise and delineate the visual objects. The solution to this task provides detailed information to understand images and is central to applications such as content-based image search, autonomous vehicles, image-editing, and smart glasses for partially-sighted people. This task is challenging to address not only because the visual objects from the same category could have a variety of appearances but also because of the need to account for contextual information across images such as edges and appearance consistency. The objective of this thesis is to bridge the gap between the pixel-based image representation in computer devices and the meaning extracted by humans. Our primary contributions are fourfold. Firstly, we propose a factorial fully-connected conditional random field that addresses the problem of jointly estimating the segmentation for both object class and visual attributes. Secondly, we embed the proposed factorial fully-connected conditional random fields framework in an interactive image segmentation system. This system allows users to refine the semantic image segmentation with verbal instructions. Thirdly, we formulate filter-based mean-field approximate inference for fully-connected conditional random fields with Gaussian pairwise potentials as a recurrent neural network. This formulation allows us to integrate fully convolutional neural networks and conditional random fields in an end-to-end trainable system. Fourthly, we show the relationship between fully-connected conditional random fields with Gaussian pairwise potentials and iterative Graph-cut: We found that fully-connected conditional random fields with Gaussian Pairwise potential implicitly model the unnormalised global colour models for foreground and background.