Auditory-visual perception of Mandarin lexical tone using 3D display

Abstract

Research on lexical tone perception has identified patterns of head movements that accompany the production of Mandarin tones. The present study used 3D presentations to investigate whether greater access to depth cues could facilitate tone discrimination in native and non-native tone language listeners (Mandarin and Australian-English). Participants completed a same/different task constructed using syllables /ma/ and /na/, spoken with the 4 Mandarin tones by two different talkers. These syllables were presented in 5 modalities: Audio Only (AO), Two-dimensional AV (2D-AV), Three-dimensional AV (3D-AV), Two-dimensional visual only (2D-VO) and Three-dimensional visual only (3D-VO). Within each modality there were two speech production modes: normal speech and clear speech. Contrary to the hypotheses, performance was similar across AO, 2D- and 3D-AV as well as normal and clear speech conditions; neither 3D presentation nor clear speech improved discrimination. However, performance on the visually salient /ma/ was better than for /na/ suggesting that additional segmental visual information facilitated performance. As expected Mandarin listeners performed better than Australian English listeners. The results highlight inter-talker variation in the production of tone and generate new avenues for study including exploring tone perception with hearing impaired listeners who may rely more on and benefit more from 3D presentation.