Language material for English audiovisual speech recognition system development

Abstract

The bi-modal speech recognition system requires a 2-sample language input for training and for testing algorithms which precisely depicts natural English speech. For the purposes of the audio-visual recordings, a training data base of 264 sentences (1730 words without repetitions; 5685 sounds) has been created. The language sample reflects vowel and consonant frequencies in natural speech. The recording material reflects both the lexical word frequencies and casual speech sound frequencies in the BNC corpus of approx. 100m words. The semantically and syntactically congruent sentences mirror the 100m-word corpus frequencies. The absolute deviation from source sound frequencies is.09% and individual vowel deviation is reduced to a level between 0.0006% (min.) and 0.009% (max.). The absolute consonant deviation is 0.006% and oscillates between 0.00002% (min.) and 0.012% (max.). Similar convergence is achieved in the language sample for testing algorithms (29 sentences; 599 sounds). The post-recording analysis involves the examination of particular articulatory settings which aid visual recognition as well as co-articulatory processes which may affect the acoustic characteristics of individual sounds. Results of bi-modal speech elements recognition employing the language material are included in the paper.