3D reconstruction of non-rigid surfaces from realistic monocular video

Abstract

A novel algorithm for recovering the 3D shape of deformable objects purely from realistic monocular video is presented in this paper. Unlike traditional non-rigid structure from motion (NRSfM) methods, which have been studied only on synthetic datasets and controlled lab environments that needs some prior constraints (such as manually segmented objects, limited rotations and occlusions, or full-length trajectories), the proposed method has been described and tested on realistic video sequences, which have been downloaded from some social networks (such as Facebook and Twitter). In order to apply NRSfM to the realistic video sequences, because of no-prior information about the scene and camera parameters, one should employ different methods that can handle a huge amount of unknown parameters (such as 3D shape and camera parameters) and deal with some other ambiguities such as incomplete segmentation and Tracking. In this paper, this goal is concerned by first proposing a novel method for completing the missing trajectories (as the most important challenge in realistic videos due to occlusions and lighting changes) and then applying a method that formulates the NRSfM as an energy minimization problem. The proposed method is evaluated on popular video segmentation datasets and its performance is compared to other available methods.