Integrating hybrid priors for face photo-sketch synthesis

Abstract

Benefiting from the advancement of deep learning techniques, face photo-sketch synthesis has witnessed significant progress in recent years. Cutting-edge methods typically treat this task as an image-to-image translation problem and train a conditional generative model to learn the mapping between two domains. However, purely parametric deep learning models often struggle to capture instance-level details due to limited training samples and tend to focus on domain-level mapping. Moreover, sketch-to-photo synthesis is more challenging than photo-to-sketch synthesis and holds greater significance in the realm of public security, but it has not been well-studied in existing methods. To address these challenges, we introduce an innovative framework that synergistically integrates parametric and non-parametric approaches, infusing facial generative priors and instance-level prior knowledge from the target domain to enrich texture detail synthesis. Specifically, our framework employs a semantic-aware network to facilitate coarse cross-domain reconstruction, thereby capturing domain-level information. Moreover, through efficient neural patch matching between the input image and multiple reference (training) samples, we can harness instance-level prior knowledge as a detailed texture representation to enhance detail fidelity. For the sketch-to-photo synthesis task, we further propose a local patch correspondence mechanism that improves the rationality of matching through local constraint. To further enhance the generation of realistic and detailed facial features, we incorporate a pre-trained StyleGAN as the decoder, leveraging its extensive facial generative priors. Additionally, we introduce the relaxed Earth Movers Distance (rEMD) loss to improve the style consistency between the generated results and the target domain. Extensive experiments show that our method achieves state-of-the-art performance on both quantitative and qualitative evaluations.