Multi-Modal Reflection Removal Using Convolutional Neural Networks

Abstract

Although color images are easily interfered by glass, depth images captured by infrared sensors are robust to reflection. In this letter, we propose multi-modal reflection removal using convolutional neural networks (CNNs). We build a multi-modal CNN for reflection removal to separate transmission from reflection using depth information. The proposed network consists of two sub-networks: image restoration and depth adaptation. Image restoration sub-network (IRN) recovers transmission layer from the input image with reflection, whereas depth adaptation sub-network (DAN) guides reflection removal of the IRN. Moreover, to extract image details for reflection removal, we present a multi-scale loss function that penalizes non-similarity for multi-scale outputs. Experimental results demonstrate that the proposed method is robust to dominant reflections and outperforms state-of-the-art methods in terms of both peak signal-to-noise ratio (PSNR) and structural similarity.