Deep joint image enhancement using convolutional neural network

Abstract

Demosaicking, denoising, and super-resolution are the critical image enhancement tasks in digital imaging pipelines, which address image artifacts. Traditionally, these tasks in the pipeline are performed by non-machine learning-based approaches, where each algorithm is performed one after the other. Nevertheless, the recent developments in convolutional neural network (CNN) based image enhancement applications have shown superior output quality. Regarding demosaicking CNNs, a comprehensive analysis that fairly compares various architectures and design features such as depth, width and loss functions is not available in the literature. Moreover, the common CNN demosaicking approach is based on training the model for a single colour filter array (CFA) pattern; although, demosaicking for multiple CFA patterns can be achieved, thanks to the capability of CNNs for addressing non-linear tasks. Further to demosaicking, a joint demosaicking, denoising, and super-resolution (JDDSR) CNN can reduce the image artifacts accumulated along the imaging pipeline, improve image quality and reduce the latency. Therefore, this research aims to develop a deep CNN that addresses these limitations by performing joint demosaicking, denoising, and super-resolution while analyzing the effects of different design features on performance. In this work, six key contributions are presented: (1) a lightweight autoencoder demosaicking CNN approach to handle multiple CFA patterns using an indexing matrix, enabling the network to adapt to different CFAs without separate algorithms, (2) the development of a generative CNN for demosaicking, which outperforms existing autoencoder-based approaches, (3) the creation of a new low-noise dataset from Adobe 5K images for use in training and evaluation, (4) a generative CNN based JDD, (5) a generative-based CNN JDDSR, and (6) a GAN-based JDDSR solution. The autoencoder CNN performs similarly to state-of-the-art networks with a compact, lightweight design. Furthermore, the indexing matrix allows CNN to learn demosaicking for multiple CFA patterns at a similar level to the single CFA trainings. Compared to autoencoder CNN, generative-based CNN performs demosaicking with fewer artifacts and generates higher-quality images. The JDD CNN presented outperformed the state-of-the-art networks by a significant margin thanks to its deep and wide architecture. The images generated using the JDDSR approach based on generative CNN and GAN include artifacts resulting in poor image quality. Hence, the performance should be improved by having a larger training set and optimized training configuration.