Audio-visual fuzzy fusion for robust speech recognition

Abstract

Improvements of robustness of speech recognition is one of the hottest topics in speech signal processing, particularly when applied within a noisy environment. Most of the research efforts focused in combining audio and visual data to implement an audiovisual speech recognition (AVSR) system. Bimodal approach demonstrated that a superior performance can be gained compared to the separate audio or visual approach. This paper proposes a fuzzy logic-based data fusion method that combines the recognition capabilities of two independent working systems namely the automatic speech recognition system (ASR) and the automatic visual recognition system (AVR). The main purpose is to boost the whole system's performance keeping the ASR separate from the AVR. This approach provides a powerful method that enables simpler data fusion at decision level rather than the more complex at data and features level. Such complexity is also lowered due to the fuzzy logic-based implementation of the data fusion engine. Preliminary experimental results confirms the proposed approach.