Adversarial Joint-Distribution Learning for Novel Class Sketch-Based Image Retrieval

Abstract

In the information retrieval task, sketch-based image retrieval (SBIR) has drawn significant attention owing to the ease with which sketches can be drawn. The existing deep learning methods for the SBIR are very unrealistic in the real scenario, and its performance reduces drastically for unseen class test examples. Recently, Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) has drawn a lot of attention due to its ability to retrieve the novel/unseen class images at test time. These methods try to project sketch features into the image domain by learning a distribution conditioned on the sketch. We propose a new framework for ZS-SBIR that models joint distribution between the sketch and image domain using a generative adversarial network. The joint distribution modeling ability of our generative model helps to reduce the domain gap between the sketches and images. Our framework helps to synthesize the novel class image features using sketch features. The generative ability of our model for the unseen/novel classes, conditioned on sketch feature, allows it to perform well on the seen as well as unseen class sketches. We conduct extensive experiments on two widely used SBIR benchmark datasets-Sketchy and Tu-Berlin and obtain significant improvement over the existing state-of-the-art. We will release the code publicly for reproducibility of results.