Automatic foreground segmentation from multi-view images with planar background

Abstract

In this paper, we present an algorithm to infer foreground segmentation from given a sequence of images. In our system, we can capture the interested object on a planar background with a handheld camera. There are two main assumptions are mentioned: 1) the region of interest appears entirely in all images; 2) the background pixels have a similar plane projective transformation, i.e., the foreground objects has different transformation from the background regions. First, we start by the structure from motion (SFM) method to get the camera calibration parameters and using the plane projective transformation with recovering feature points on the background plane to calculate the homography matrices between each frame. For the main foreground segmentation step, according to the two assumptions mentioned above, we define an energy function with the color distance set in a segmentation based and minimize the energy using the belief propagation (BP) method. After we obtain a series of silhouette maps and theirs camera projective matrices, we apply the image based visual hull (IBVH) method and Poisson surface reconstruction method to rebuild the 3D model of our interested object.