Stage-Aware Feature Alignment Network for Real-Time Semantic Segmentation of Street Scenes

Abstract

Over the past few years, deep convolutional neural network-based methods have\nmade great progress in semantic segmentation of street scenes. Some recent\nmethods align feature maps to alleviate the semantic gap between them and\nachieve high segmentation accuracy. However, they usually adopt the feature\nalignment modules with the same network configuration in the decoder and thus\nignore the different roles of stages of the decoder during feature aggregation,\nleading to a complex decoder structure. Such a manner greatly affects the\ninference speed. In this paper, we present a novel Stage-aware Feature\nAlignment Network (SFANet) based on the encoder-decoder structure for real-time\nsemantic segmentation of street scenes. Specifically, a Stage-aware Feature\nAlignment module (SFA) is proposed to align and aggregate two adjacent levels\nof feature maps effectively. In the SFA, by taking into account the unique role\nof each stage in the decoder, a novel stage-aware Feature Enhancement Block\n(FEB) is designed to enhance spatial details and contextual information of\nfeature maps from the encoder. In this way, we are able to address the\nmisalignment problem with a very simple and efficient multi-branch decoder\nstructure. Moreover, an auxiliary training strategy is developed to explicitly\nalleviate the multi-scale object problem without bringing additional\ncomputational costs during the inference phase. Experimental results show that\nthe proposed SFANet exhibits a good balance between accuracy and speed for\nreal-time semantic segmentation of street scenes. In particular, based on\nResNet-18, SFANet respectively obtains 78.1% and 74.7% mean of class-wise\nIntersection-over-Union (mIoU) at inference speeds of 37 FPS and 96 FPS on the\nchallenging Cityscapes and CamVid test datasets by using only a single GTX\n1080Ti GPU.\n