Image Inpainting Using Lightweight Transformer Neural Network Based on Channel Attention

Abstract

Image inpainting is an important image processing application in repairing images with damaged or undesirable contents. While traditional approaches based on pixel or patch matching lack the ability to generate novel contents for the missing regions in an image, generative neural network has shown its potential in making up meaningful contents for the masked areas. On the other hand, to bridge the gaps between visible and missing regions, both traditional and deep-learning based approaches utilize self-similarity property of an image. In generative learning, this self-similarity property is explored using attention mechanisms. Since structural information may have multiple meanings depending on neighborhood context but a single attention mechanism can only provide one interpretation, applying transformer architecture with multiple attentions in image inpainting should further improve the final results by giving multiple interpretations to self-similarity information. The limitation of applying transformer is that complexity of an attention mechanism is proportional to the square of the number of pixels in an image. This complexity could grow exponentially and become infeasible for capacity-limited hardware if image resolution is high. In this paper, we propose to replace spatial-attention based transformer with one based on channel attention. Thereby, the complexity is reduced to the square of the number of channels which does not depend on spatial resolution and can be easily controlled as a network parameter. To remedy the loss of spatial information because of channel attention, a spatial gating mechanism is added after the channel-attention transformer. After this new lightweight transformer architecture generates the coarse result, an attention-based refinement network is used to generate the final result. Experimental results show that the new transformer architecture can produce inpainting results comparable to state-of-the-art approaches with far less network parameters.