Audio-visual multi-person tracking for active robot perception

Abstract

Integration of signals acquired by the sensors of different modalities is known to enhance robot perception. This work presents a multimodal system consisting of audition, vision and motion modalities for human tracking with navigational tasks in a real environment having obstacles and noises affecting robot perception. The main goal of the system is to track the human who interacts with the robot. In the audition modality, multiple signal classification based on generalized eigenvalue decomposition method is utilized for sound source localization and the complementary vision modality incorporates face detection including a method combining feature-based and color-based techniques and tracking facilities to provide continuity in tracking the human. The other perceptive ability of the robot is to detect obstacles while tracking and approaching the human and to avoid them without losing the target. The aim of the successive motion modality is to enable a robot having intelligent and human-like behaviors by using the output from the sensor fusion framework and obstacle detection. To evaluate the performances of robot perception and behaviors, a mobile robot is utilized in both single and multi-person experiments in a real environment with obstacles.