Weakly Supervised Few-Shot and Zero-Shot Semantic Segmentation with Mean Instance Aware Prompt Learning

Abstract

The existing Zero-Shot Segmentation (ZSS) and Few-Shot Segmentation (FSS) methods utilize fully supervised pixel-labeled seen classes to segment unseen classes. Pixel-level labels are hard to obtain, and using weak supervision in the form of inexpensive image labels is often more practical. To this end, we propose a novel unified weakly supervised Zero-Shot and Few-Shot semantic segmentation pipeline that can perform ZSS and FSS on novel classes without using pixel-level labels for either the base (seen) or the novel (unseen) classes. We propose Mean Instance Aware Prompt based Network (MIAPNet), a novel language-guided segmentation pipeline that i) learns context vectors with batch aggregates (mean) to map class prompts to image features and ii) decouples weak ZSS/FSS into weak semantic segmentation and Zero-Shot segmentation. MIAPNet beats existing methods for weak generalized ZSS and weak FSS by 39 and 3 mIOU points respectively on PASCAL VOC and weak FSS by 5 mIOU points on MS COCO.