Multimodal human-computer interaction

Abstract

While human-to-human communication takes advantage of an abundance of information and cues, human-computer interaction is limited to only a few input modalities (usually only keyboard and mouse) and provides little flexibility as to choice of communication modality. In this paper, we present an overview of a family of research projects we are undertaking at Carnegie Mellon and Karlsruhe University to overcome some of these human-computer communication barriers. Multimodal interfaces are to include not only typing, but speech, lip-reading, eye-tracking, face recognition and tracking, and gesture and handwriting recognition. Initial experiments aimed at exploiting the complementary nature of these alternate modalities in interpreting user intent in a user interface are discussed. KEYWORDS: Multiple modalities, multimodal interface, speech recognition, lip-reading, eye-tracking, gesture recognition, handwriting recognition. 1. INTRODUCTION With multimedia workstations and high-speed dat...