Discriminative stream‐weight training for mandarin audio‐visual speech recognition

Abstract

Abstract In a large vocabulary audio‐visual speech recognition system, to efficiently improve the robustness of the system and reduce the word error rate, two discriminative stream‐weight training methods are provided. The state‐dependent stream weights are trained based on lattice rescoring by the minimum phone error and boosted maximum mutual information using the extended Baum Welch algorithm respectively. Experimental results show considerable error reductions have been achieved by the proposed methods over those using global stream weights. It is also shown that these methods provide better results than the minimum classification error based stream weight training methods.