Transformer-Based Neural Texture Synthesis and Style Transfer

Abstract

Texture modeling has been a research hotspot for long, containing topics of neural texture synthesis and neural style transfer, have gained significant attention from both industry and academia. Prior arts prevalently utilized Convolutional Neural Networks as basis for performing neural texture synthesis and neural style transfer tasks, however, they hardly explore other deep neural architectures. Is convolutional network a must for texture modeling tasks? In this work, we explore this problem by introducing a novel framework along with novel optimization objectives for Transformer-based texture synthesis and style transfer. We proposed a novel texture description metric which works well in the feature space of Transformers, and more lightweight than Gram-based texture descriptors. We also proposed pixel-level and patch-level smoothing regularizations to help the generative process. Our approach shows significant improvement upon the baseline and generates favorable results, showing that we can make use of Transformers' long-range dependencies to perform texture modeling and style transfer tasks without the help of convolutional layers.