User performance improvement via multimodal interface fusion augmentation

Abstract

The functional role of a data fusion system is to provide timely and accurate information to the user. At the lowest level, the fusion system filters relevant incoming data and at the highest level, the fusion system presents actionable information to the user. In many cases, the fusion system design is dependent on user capabilities and knowledge. Thus, only presenting the fused data might not enhance operator perjormance, while presenting all the multimodal information might overload the operators cognitive capabilities. In this paper, we use the JDL-User model to design an interface which utilizes image fusion and multimodal characteristics to enhance user performance. The design allows for the user to passively monitor and actively interact with the fusion system with controls for image overlays, queries and audio warnings to support unattended areas, and an eye tracker to improve attention. Implemented in JA VA, we present a tracking display design that utilizes a multimodal interface to (I) take advantage of the human's ability to fuse muliisensory information, (2) facilitate situational awareness, and (3) enhance the users cognitive task analysis.